System iNetwork Head Nav Subscribe Log In Contact Us Advertise User login Username: * Password: * Request new password

Search Primary links Forums Archives Code Blogs Podcasts Webcasts e-Learning Guides Newsletters About Us Contact Us About the Network Tech Editor Profiles Editorial Calendar Writers Kit Advertise Join Network Categories RPG Programming Other Languages Application Development Application Modernization Database/SQL Availability Security Systems Management Networking IT Mgmt/Careers Site Links Solutions Buyer's Guide UK Centre Jobs Home Content Introduction to Pointers in RPG Article ID: 55776Posted October 25th, 2007 in RPG Programming Application Develo pment By:Scott Klement Q: My RPG program needs to receive an XML document from WebSphere MQ. It can be longer than 65535 characters, which is RPG's maximum. I've heard that it's possi ble to handle large data with pointers, can you tell me how? A: I can't help you with WebSphere MQ, but I can explain how pointers work, and how you can use them to handle data that's much longer than 65535 characters. Th is article is an introduction to pointers, aimed at RPG programmers who have nev er used them before. Variables and Memory In RPG, we typically store data in variables, but of course there are limits to how large a variable can be. But, what is a variable? Computers don't really understand variables -- not dire ctly, anyway. A variable is something that programming languages supply to make things easier for humans like you and me to understand. Computers actually store the data for variables in memory, and refer to it by address.

For example, suppose you write the following code: D buff s 65535A When the RPG compiler compiles the source, it inserts instructions into your pro gram to ask the operating system for 65535 bytes of memory. The operating system manages the memory because there are lots of programs running on your system al l at once, and they all need to share the same memory pools. So the compiler gen erates code to ask the OS for memory for each variable you define. When this hap pens, the operating system finds a spot where there's 65535 bytes available, and returns the address of the memory in the computer's main storage. It's that space in memory, not the variable, that the computer actually stores a nd retrieves data from. On most computer systems, a memory address is just a number. If you have 1 GB of RAM, the memory address will be a number between one and one billion. The purpo se of the address is to identify a particular byte in the computer's memory. On i5/OS, addresses are more complicated than that because the single-level store p aradigm of the operating system makes memory work differently -- but explaining how that works is well beyond the scope of this article. For the purposes of poi nters and memory manipulation in RPG, it doesn't matter -- just think of a memor y address as a number that identifies a particular spot in your computer's memor y. Pointers A pointer is a type of variable. You're already familiar with other types of var iables, such as alphanumeric variables (for example, the 65535A one above) that store character strings, numeric variables (such as a zoned or packed number) th at store numbers, date variables (type D in RPG) that store dates, and so forth. There's also a type of variable that's designed to store memory addresses. Afte r all, if our program works with memory addresses, it needs a place to store the m. An ordinary numeric variable will not do -- we need to store it in a variable designed for storing memory addresses. A "pointer" variable is a variable designed to store memory addresses. It works like any other variable (in fact, it works very much like a numeric variable). T he only difference is that it's designed for a special type of number: a memory address. In RPG, pointers are designated by specifying a data type of *. For example: D bufptr s * In RPG, the initial value of a pointer will be a special value of *NULL. *NULL m eans that the pointer doesn't contain a memory address. It's set to nothing. It' s not currently pointing to any spot in memory. However, you can use the %ADDR() built-in function to get the memory address of another variable, and you can set the pointer to point to that same memory addre ss. You can also use the %ALLOC built-in function to ask the operating system fo r some memory, and the operating system will return a memory address which you c an then store in your pointer. Since a pointer refers to a particular place in memory, pointers are sometimes c alled a "reference" to memory. For example, when you pass a parameter from one p rogram to another on i5/OS, the parameters are "passed by reference," which mean s that under the covers the only thing that gets passed is the value of a pointe r -- a memory address. Based Variables Normally, when your program begins, it asks the operating system for a place in

memory to store its data. When you store data into your variable, it's actually just setting the values of the bytes in that particular location in the computer 's memory. When you read the contents of a variable, it's actually just reading the bytes that are stored at that particular location in memory. That's how vari ables work. But, suppose you don't want RPG to automatically ask the operating s ystem for memory? Suppose you want to control all of this for yourself? In that case, you use a BASED variable. You define a BASED variable in RPG by coding the BASED keyword on the D-spec. Fo r example: D buff s 65535a based(bufptr) In the above statement, the variable named buff will not have any memory reserve d automatically when the program starts. Instead, it's based on the pointer name d bufptr. It's up to you to control where bufptr points, and therefore to contro l which space in the computer's memory will be read when you access the contents of buff, or which space in memory will be updated if you change the value of bu ff. You control that space in memory by manipulating the pointer that the variab le is based on -- bufptr in this example. You can set bufptr to point to anywher e in the computer's memory, and buff will then "view" that space. This act of accessing the memory that a pointer points to is sometimes called "d e-referencing" the pointer. Let's look at a simple example program (this is a complete program, you can try it on your computer if you like.) D D D D FIELD1 p_Field2 FIELD2 FIELD3 s s s s 10A * inz(*null) 1A based(p_Field2) 7P 0 inz(1234567)

/free Field1 = 'SS/400'; p_Field2 = %addr(Field1); Field2 = 'A'; // (1) Field1 now contains 'AS/400' dsply Field1; p_Field2 = %addr(Field1) + 5; // (2) Field2 now contains '0' dsply Field2; Field1 = 'System o5'; // (3) Field2 now contains 'm' dsply Field2; p_Field2 = p_Field2 + 2; Field2 = 'i'; // (4) Field1 now contains 'System i5' dsply Field1; p_Field2 = %addr(Field3) + (%size(Field3) - 1); Field2 = x'0d';

// (5) Field3 now contains '-1234560' dsply Field3; *inlr = *on; /end-free At the start, this program asks the operating system for memory. It asks for 10 bytes for FIELD1, and 4 bytes for FIELD3 (Field3 is packed, and therefore can st ore a 7-digit number in 4 bytes of memory.) It does NOT ask for memory for FIELD 2, because FIELD2 is a based field. FIELD2 is based on a pointer named p_Field2. In this example, I've declared p_Field2 on it's own D-spec, and set it to *NULL with the INZ keyword, so it doesn't point anywhere in memory. It might be worth noting that if I forgot to create a D-spec for the p_field2 po inter, it wouldn't have mattered. RPG would've automatically created a pointer n amed p_Field2 because it's referenced in a BASED() keyword. So you don't actuall y have to use a D-spec for p_Field2 if you don't want to, RPG will do it automat ically. Also, RPG sets pointers to *NULL by default, so I didn't really need the INZ, either. At the start of my calcs, I set FIELD1 to the value 'SS/400'. Again, the operati ng system reserved memory -- 10 bytes -- for FIELD1 and returned an address to m y RPG program. The compiler generated code that runs under the covers and manage s this memory for me. Let's say for example that the operating system returned t he number 1000 for the memory address. In the real world, the number would be mu ch more complicated than that... it would be something like x'8000000000000000CF 2C48D1E30740B0', but just to keep the math simple, let's say it returned 1000. That means that the FIELD1 variable occupies positions 1000-1009 of my computer' s memory. RPG will automatically set FIELD1 to blanks, so positions 1000-1009 al l contain x'40' when my calcs start. Then my program changes position 1000 to 'S ', position 1001 to 'S', position 1002 to '/', position 1003 to '4', position 10 04 to '0' and position 1005 to '0'. I hope you get the idea. Then, I set p_Field2 to %addr(field1). This gets that address (the number 1000) where RPG was storing FIELD1 and stores that value in p_Field2. So p_Field2 now contains the value 1000. Because FIELD2 is based on p_Field2, it now will view t he memory at address 1000 -- so right after the line that sets p_field2 to %addr (Field1), the Field2 variable will contain 'S', since that's what happens to be stored at position 1000. When I change Field2 to be an 'A', it changes position 1000 of the computer's memory, which results in FIELD1 now having the value 'AS/ 400'. The first DSPLY statement (marked in red) will display 'AS/400' Then, it does a p_field2 = %addr(Field1) + 5. If you think about it, if the addr ess of Field1 was 1000, and you add 5, you get 1005. So p_field2 = 1005. In posi tion 1005 was the number 0, the last character of the string 'AS/400'. Keep in m ind that changing this pointer doesn't change the memory it points to. When I ch anged p_Field2, it changed Field2 to have a value of '0' because it changed the place in memory that it's viewing. Position 1000 still contains 'A', and FIELD1 still contains 'AS/400'. The second DSPLY will display '0' since that's whats in memory at position 1005. Next I change FIELD1 to contain the string 'System o5'. This changes the bytes a t positions 1000-1008 to contain 1000=S, 1001=y, 1002=s, and so on. Position 100 5 now contains 'm', so even though I didn't change FIELD2, and I didn't change p _Field2, the value of FIELD2 still changed. FIELD2 now contains 'm' because that 's what's stored in position 1005 of the computer's memory. The third DSPLY will display 'm'. When I do p_Field2 = p_Field2 + 2, I change the pointer again. It was previously

set to 1005, but now it'll be set to 1007. Since position 1007 is the 'o', FIEL D2 will now contain 'o'. When I change FIELD2 to have a value of 'i', it changes position 1007 in memory to have the value 'i', and now FIELD1 will read 'System i5'. At the fourth DSPLY, it will display 'System i5'. Pointers work on non-character variables as well -- they're operating on memory directly, and all data is stored in memory. Field3 is a packed number that needs 4 bytes of memory. Again, the system would've reserved memory for it when the p rogram started -- lets say it's stored in memory at location 1500 (again, just t o make the math easy) so it occupies memory locations 1500-1503. The INZ on the D-spec has set the value to 1234567, which means that position 1500=x'12', posit ion 1501 = x'34', position 1502=x'56', and position 1503 is x'7F'. The F in a pa cked number means that the number is positive. If it were set to 'D', it'd be a negative number. The %SIZE BIF gets the size -- in bytes -- of a field. So %SIZE(Field3) = 4. The expression p_Field2 = %addr(FIELD3) + (%size(Field3)-1), what happens? Well, 4 - 1 = 3. %addr(FIELD3) = 1500, so 1500+3 = 1503. I've just set p_field2 to 1503. That means that Field2 will have a value of x'7F' since that's what was stored in location 1503. When I change Field2 to x'0D' I've changed position 1503 to x' 0d', and that changes the last digit of field2 to zero. It also changes the sign from F to D, which makes it negative. So the last DSPLY of the program will dis play -1234560 since that's the value that's now in FIELD3. Hopefully that example program gives you an idea of the following: %ADDR() gives the address of a variable in memory, so that it can be stored in a pointer. You can add or subtract from an address in memory to get another address that's near by. You can base a variable on a pointer, so that the variable occupies the place in memory that the pointer points to. You can change that based variable to change the memory. You can move the based variable to different places in memory by changing the po inter. Dynamic Memory Allocation If you like, you can ask the operating system to reserve a spot in memory for yo u. This is done by calling the %ALLOC BIF (or the ALLOC op-code, which does exac tly the same thing). This is often useful to RPG programmers because it lets you reserve up to 16 MB of memory, which is much, much larger than the 65535 limit we currently have for strings. For example, consider the following example: D bufptr D buflen D buff s s s * 10i 0 32768a based(p_buff)

/free // ask the operating system for one megabyte // of memory. buflen = %size(Buff) * 32; bufptr = %alloc(buflen); In this example, buff is 32768 bytes long, so %size(Buff) * 32 will be 1 MB. (ap

prox 1 million bytes of memory). So the %alloc() BIF is used to ask the operatin g system for 1 megabyte of memory. It will return the address in memory where th at 1 MB has been reserved. I store that into the bufptr pointer. Now, it's important to understand that this 1 MB of memory will NOT be automatic ally set to blanks for me like an RPG variable would be. Indeed, I do not know w hat the memory will be set to. It might be set to x'00' (this'll be the most com mon) but it might also have "garbage" in it -- data left over from something els e. I can't rely on that memory to be set to any particular value. If I want to set it to blanks, all I have to do is base a field on the pointer, and then set that field to blanks. Easy right? Well, yes -- it WOULD be that eas y, except that you can't define a field that's 1 MB long in RPG. If a define a f ield that's 65555 long, I'd only set the first 65535 bytes to blanks, and the re maining space would still be uninitialized. But remember, you can change the pointer of a based field to make it view a diff erent space in memory. So with a loop and a little pointer math, you can set the whole thing to blanks, you just have to do it in smaller chunks. Notice in the code where I demonstrated %alloc(), I also had a 32k field named "buff" that was based on p_Buff. I didn't include a D-spec to define p_Buff, but that's okay; R PG will automatically declared p_Buff as a pointer (since it's used in a BASED k eyword) and it'll automatically initialize it to *NULL. So my preceding code sni ppet actually had two pointers declared, bufptr is a pointer, and was used as th e result of the %alloc() BIF, and p_Buff is also a pointer, but I didn't use it in the preceding code snippet. But, there was a method to my madness! I'll use B uff and p_Buff now to clear out the 1 MB of memory, setting it all to blanks: // blank out the entire megabyte for x = 0 to 31; p_buff = bufptr + (x * %size(buff)); buff = *blanks; endfor; Since buff is 32768 bytes long, %size() buf will return 32768. The first time th rough this loop, x=0, and zero times anything results in zero, so p_buff is set to bufptr (the start of my 1 megabyte) plus 0. When I set buff = *Blanks, it set s the first 32k of my 1 MB to blanks. The second time through, x=1, so p_buff wi ll be set to the space in memory that's 32768 bytes later than bufptr. Now that second chunk of 32k is set to blanks. This continues 32 times (since a loop from 0 to 31 is run 32 times) until the entire 1 megabyte has been set to *blanks. You do have to be very careful when doing this sort of thing. In this example, m y program deliberately declared buflen to be an even multiple of the size of buf f. When I ran through the loop, I was careful to make sure that I only accessed memory that's within that allocation. Neither RPG nor the operating system will protect you against mistakes -- if I had accidentally made my loop run from 1 to 32 (instead of 0 to 31) then on the last time through the loop, it would've mul tiplied 32768 by 32 and added it to bufptr. This would've resulted in accessing 32k of memory that comes immediately AFTER my allocation, and I would've blanked out whatever data happened to be stored in memory in the memory addresses that are adjacent to mine in memory. I did not reserve that space to my program so th e operating system might be using it for something else -- it might happen to bl ank out other variables in my program, or variables used by a DIFFERENT program, or even variables used internally by the operating system (that's unlikely, but possible). Perhaps the most likely situation is that it would blank out an unus ed portion of the system's memory. The problem is, there's a chance that I might blank out something different every time I run my program because the operating system can potentially give me a different location every time! So I might test my program, and everything works well because (by some stroke of luck) I happen

to be blanking out a space in memory that nothing else on the system is using. So, everyone's happy -- then I might put it into production, and it might work w ell for a year, no problems... then someone happens to run it in a different cir cumstance, and an unrelated program on my system starts crashing, and I don't kn ow why, because everything had been working so well prior to that! And it's beca use my program was blanking out memory that it should've blanked out.... but it might take a very long time to discover this, because bugs like this are NOT eas y to find. So you do have to be VERY careful when working with memory like this. You have to give a lot of thought to it, and be sure that your code only access es the memory that's been reserved to it. Anyway... now that I have 1 MB of blank memory in my program, I should be able t o (theoretically, anyway) receive up to 1 MB from MQ by using that memory for th e MQGET buffer. // receive data via MQ // buflen = 1 megabyte // bufptr = 1 megabyte of memory that the OS reserved // for this program to use. // this means you can receive up to 1 megabyte from MQ, // (assuming that MQ supports it.) mqget( hconn: hobj: mqmd: mqgmo: buflen: bufptr : msglen: CCode: Reason ); I should warn you, however, that I know absolutely nothing about WebSphere MQ so I have no clue if I'm coding that statement correctly. I'd really appreciate fe edback from readers about whether this code works as expected. It's worth mentioning, however, that this technique is not limited to the WebSph ere MQ APIs. It works well with most i5/OS APIs, including the socket APIs, the HTTP server APIs (such as QtmhRdStin() and QtmhWrStout()), and the IFS APIs. All these APIs are used frequently when working with XML. Saving the Data to the IFS Now that I have received a document from MQ, I can (theoretically) parse it with the XML parser. Keep in mind, however, that the XML-INTO op-code cannot get its input from a pointer. Expat can, however, if you're willing to use that instead . If you want to parse this data with XML-INTO or XML-SAX, you're going to have to save it to a file in the IFS and then tell XML-INTO or XML-SAX to get its inp ut from the file. i5/OS comes with an API named tmpnam() that will give you a temporary IFS filena me. It also comes with APIs for reading/writing IFS files. It's beyond the scope of this article to explain them, but I put links at the end of the article to s ome resources where you can go to learn more about them. H BNDDIR('QC2LE') D tmpnam PR * extproc('_C_IFS_tmpnam') D string 39A options(*omit) D tempfile s 45a varying D fd s 10i 0 . . . tempfile = %str( tmpnam(*omit) ) + '.xml'; fd = open( tempfile: O_WRONLY+O_CREAT+O_EXCL+O_CCSID : S_IRUSR + S_IWUSR : 1208 );

if (fd = -1); // open() failed. endif; callp write(fd: bufptr: msglen); callp close(fd); dealloc bufptr; bufptr = *null; p_Buff = *null; The code from this code snippet would run immediately after the MQGET() API has returned the XML to my program. The tmpnam() API is part of the ILE C runtime an d requires you to bind your program to BNDDIR('QC2LE') so that the program will be able to find the API on your system. I prefer to code it in the H-spec as sho wn in the example, but you can also use the BNDDIR keyword on the CRTBNDRPG, CRT PGM or CRTSRVPGM commands if you like. I like the H-spec better because I can co de it once, and don't have to remember to type it each time I compile my program . The tmpnam() API gives me a unique, temporary filename in the IFS. It makes sure that there isn't already a file with this name, so I don't have to worry about someone else using the same filename. In my code, I've added '.xml' to the end o f the filename so that when I see it in the IFS, I'll know what type of data is stored in it (this isn't really necessary, but it can make things easier, especi ally if I want to look at the file from Windows). The open() API will open this IFS file for me and let me write data to it. I've told it to open the file for writing only (O_WRONLY) and create the file if it d oesn't exist (O_CREAT). The O_EXCL flag means that if it does exist, I want the API to return an error. It should never exist when this code is run, so O_EXCL i sn't really necessary, just an extra safeguard. I don't want to clobber someone elses file! I've also told the open() API that I want to mark the file with a CC SID. The XML-INTO/XML-SAX op-codes will use this CCSID to determine how to trans late the data in the file (is the data in ASCII? Unicode? EBCDIC?). I've specifi ed 1208, which is UTF-8. You should change this to the value that makes most sen se in your environment. For example, if you expect your XML document to be ISO-8 859-1, then you should put 819 here. If you expect your document to be UTF-16, y ou should put 1200 here. If you expect UCS-2, you should put 13488 here. Only le ave it set to 1208 if you expect UTF-8 data. Unfortunately, RPG's XML parser does not have the ability to figure out the enco ding from the contents of the file, so you do have to use the correct CCSID. Exp at, as well as most other XML parsers, do not have this problem, in which case t he CCSID is not so critical. If the XML data can be received in any arbitrary en coding (meaning that you don't know ahead of time what the encoding will be), yo u'll have to write a routine to analyze the XML and determine the encoding if yo u plan to use XML-INTO or XML-SAX. Again, with most other XML parsers you won't have to worry about that, since they already have the ability to "figure it out" from the document itself. The S_IRUSR + S_IWUSR flags give me (or whomever runs my program and creates the file) both read and write permission to the file. I do not grant any permission s to anyone else -- this is, after all, just a temporary file. The last parameter is that CCSID (1208 for UTF-8) that I referred to above. Then I call the write() API and pass my pointer and length of data to the file. The system will write the data to the IFS as is (it's not translated in this exa mple, since I didn't open the IFS file in text data mode).

The close() API closes the IFS file. Freeing Up Allocated Memory When I'm done with the memory that I asked the operating system to reserve (or " allocate"), I need to use the DEALLOC op-code to tell the operating system that it's able to use the memory for other things again: DEALLOC bufptr; If I forget to DEALLOC the memory, it will NOT be automatically freed when my pr ogram ends. Not even if my program ends with *INLR=*ON -- the memory will still be reserved and the operating system will not be able to use it for other things . It's important to remember to call DEALLOC so that the memory is cleaned up. If I somehow fail to call DEALLOC, the memory will remain allocated until the ac tivation group is reclaimed, or until my job ends (i.e., when a batch job is end ed, or when the user signs off of an interactive job.) In the "writing to the IFS" example above, you see that I used the DEALLOC op-co de to free up the memory after the data was written to the IFS. That's because I no longer need the memory in my program -- the XML op-codes will be reading the temporary IFS file and not using the pointer from my program. So, if I no longe r need it, I might as well de-allocate it and let the operating system use it fo r other things. Once I've deallocated it, it's important that my program no longer tries to read /write that area of memory. So I set bufptr and p_buff to *null to make sure tha t my program doesn't accidentally try to refer to them again. Parsing the XMLNow that the data is in the IFS file, I'm ready to ask RPG to par se it, so I'll do something like this: XML-INTO MyDs %XML(tempfile : 'doc=file allowmissing=yes case=any'); unlink(tempfile); XML-INTO op-code specifies the temporary IFS file (using the "tempfile" variable ) as the source to read the data from. The fact that I've coded 'doc=file' will ensure that it reads a file rather than trying to get it from a string in my pro gram. Once again, the reason I can't use a string is becuse XML-INTO doesn't kno w how to read data from a pointer, and since RPG's strings can't store data that 's more than 64k, I needed to put it into a temporary IFS file (which can potent ially store gigabytes or even terabytes of data). Finally, I call the unlink() API. The unlink() API is equivalent to the RMVLNK C L command, and is how you delete a file from the IFS. All I've done with the unl ink() API is deleted the temporary file I've created. Technically unlink() doesn't DIRECTLY delete the file. It removes a link to the file. You see, in the IFS, it's possible to have a single file that's listed wit h (potentially) many different file names in many different IFS directories. Eac h of these file names is called a "link" to the file. When there are no links to a file, and no program has the file open, the operating system will automatical ly the file itself. However, because I just created this file, there aren't any other links to it, a nd because I already closed it with the close() API, nobody should have it open. Therefore, when I call unlink(), the file will be deleted. If by some weird cha nce someone on the system does open the file (perhaps someone with *ALLOBJ is br owsing the IFS, and happens to display my file on the screen), it will be delete d as soon as they're done with it (which is exactly what I want to happen).

For more information on pointers & memory allocations, there's a little tutorial on my web site called "Fun with RPG Pointers". I wrote it a long time ago, so i t might be a bit outdated, but perhaps you'll find it interesting: http://www.scottklement.com/rpg At the preceding link, there's also an e-book called "Working with the IFS in RP G IV" that talks about using the IFS APIs. It's written for V4 level code (I thi nk I was using V4R5 at the time). You might find it to be a good way to learn th e IFS APIs. More recently, I wrote a 7-part series for System iNEWS magazine about using the IFS APIs. ( It even won two awards :-) ) The following are links to the article s, they do require a ProVIP membership to view: Introduction to Stream Files, November 2004, article ID 19312 A Text File Primer, December 2004, article ID 19473 Text Files in the World, January 2005, article ID 19626 Binary Stream Files, February 2005, article ID 19751 Getting Information About Your Files, May 2005, article ID 20050 Working with Links, June 2005, article ID 20141 Working with Directories, September 2005, article ID 20235 Information about parsing XML with Expat can be found at the following link: http://www.scottklement.com/expat/ Information about parsing XML with XML-INTO and XML-SAX can be found in the Info rmation Center at the following links: http://publib.boulder.ibm.com/infocenter/iseries/v5r4/topic/books_web/c092507616 8.htm http://publib.boulder.ibm.com/infocenter/iseries/v5r4/topic/books_web/c092508654 2.htm Bookmark/Search this post with: Login to post comments Printer-friendly version Related Links How to Create a Word Document in RPG Podcast: Jon Paris Gives Thoughts on Open Access Great New Features in Easy400.net's Excel Generator Scanning a Slew of Spaces, Part 2 Read a CSV File with an SQL Statement ProVIP Sponsors ProVIP Sponsors Featured LinksSponsored Links Footer Site Links Home Subscribe Now Advertise Contact Us Feedback Terms of Use Trademarks Privacy Statement 2011 Penton Media, Inc.