Interfacing 100T Ethernet And Embedded Systems
Circuit Cellar Magazine (Nov 2004)
By Eddie Insam
Copyright 2004 Eddie Insam edinsam@eix.co.uk
TITLE
Interfacing 100Mbps Ethernet And Embedded Systems
STRAP
Fast Ethernet and small micro controllers do not mix, or so they say. However, adding full speed 100Mbps Ethernet to an embedded system is not an impossible to do. Eddie Insam will show how this can be done with a little help from some supporting hardware.
INTRO
Another article about Ethernet and embedded systems? Well, yes but here we are talking about 100Mbps. Yes, the fast version, not the 10Mbps sloggers usually associated with small embedded systems. Who wants high speed Ethernet anyway? I thought you might ask. If you need to feed data from a fast source such as a CCD camera, voice or high speed data converter you will need to use some high speed method of getting it into your PC. Firewire, USB2 are possibilities, but Ethernet remains one of the comfiest methods for packing fast data into a PC. It also means your peripheral can be sited a long way away, something you just can't do with Firewire or USB.
Mind you, it is difficult enough to get a 10Mbps Ethernet controller working anywhere near full speed when paired with a small micro. These cronies can take an eternity to move data in and out of the line, and they do it mostly one byte at a time. Slap in a faster microcontroller? It won't necessarily help. You will need a pretty powerful 32 chipper plus a good helping of side IC condiments before anybody notices the difference. This article is about modesty anyway. How can we stay below the clouds and still get the performance by using relatively cheap hardware?
Chances are that you will have been peeking at the captions before reading this. You may have noticed that FPGAs are mentioned. Aha, that's how you do it. You slap in a fancy FPGA to do all the fast work, and let the micro fulfil its existence by doing the boring tasks such as housekeeping and flashing the LEDs. Well that's the gist. But then, as easy as said, how do we design such a system?
I'll bet that more than one FPGA enthusiast out there will suggest that we need nothing more than just the one whacking big FPGA. Get rid of the micro, get rid of all the other ICs, put everything inside the FPGA. Get rid of the Ethernet controller too, that will fit inside the FPGA no problem, might as well add the kitchen sink too. Of course, he may just forget to tell us that we shall need to spend the rest of our youthful years designing such a system.
Returning to real life, one wonders whether there is a line to be drawn. How much should we put in the FPGA? Do we need a supporting micro? How much should it be doing? Do we really need an Ethernet controller? Won't that mean the FPGA will be doing very little? And the bottom line question to cap it all: how can I do this project with doing as little work as possible?
Those were the decisions I was fumbling with when designing a specialist high speed interface for a PC. In the end, my old friend, the wizard Gandalf showed me the way. Well, he wasn't really a wizard, just a colleague who used to work for Gandalf, at least he had a greyish beard. He impressed upon me that one of the karmas of life enjoyment is to use as much existing off-the shelf technology as possible, even if it costs a bit more. This brilliant piece of logic assumes of course we are not slaving for a mass market toy manufacturer where the opposite applies and life appreciation issues are not an option.
Delving on this pretty obvious re-inventing the wheel type of advice, I decided to find the best route. Not so much in terms of minimum parts cost, but in terms of value for money and peace of mind.
Seasoned electronic designers and micro software designers can look at a project and within a short time visualise the amount of effort required for development. Not so with FPGAs, especially where you may be hitting their hidden performance walls. Development and testing time for these things can be pretty heavy in terms of manpower and general aggravation. On a par-to-par comparative basis, software development for a micro controller is more result-effective than low level development for an FPGA. So I should resist the geek's temptation to put everything under the sun in the FPGA, and accept old Gandalf's advice.
The FPGA only needs to do what is minimally required of it to do. All the unused ICs and facilities can go unconnected and unused. For this project, only the very fast payload data transfers need to be implemented using fast logic, and this is where the FPGA comes in. The rest of the job is handled with an off the shelf micro and a standard Ethernet Controller.
FIG 1 Caption: The Prototype layout. Note the Ethernet LAN controller at the top, and the FPGA around the middle of the board.
Fig2 Caption: General block diagram. Fast data is steered away from the micro into the FPGA internals for further processing. The multiplexed arrangement conveniently allows the FPGA to be masked off the system during development.
OK, so we have a microcontroller, an Ethernet interface chip and a FPGA. You can see the general prototype PCB layout in FIG1, The general block arrangement is shown in FIG2.
Note how the FPGA sits between the micro and the Ethernet Controller. Within the FPGA, a data multiplexer steers control to the Ethernet chip between the micro and the rest of the processing subsystem within the FPGA. This is a very convenient arrangement, the micro can drive the Ethernet chip directly for initialisation purposes and for handling low speed protocols such as ARP and PING. The multiplexer switch is flipped over to the FPGA only when there is fast data that needs to be moved across. This data is output in 16 bit parallel form directly to the external hardware device (in this case a fast D/A converter). A similar kind of logic applies to the transmission of data, input data is shifted to the Ethernet chip when it is ready to transmit the payload, and after the micro has pre-initialised all its registers. The FPGA job is very simple, however it still needs to perform some semi-skilled jobs such as the calculation of internet checksums on the fly, so the FPGA is not completely dumb. In fact, in this design I used a custom CPU core to perform these functions, more on this later.
This multiplexed arrangement is also neat for development. The FPGA can simply be programmed out of the way, in other words, I can write a simple FPGA program to connect the micro pins directly across to the Ethernet controller chip. This means I can develop software, test the board etc, without the added complexity of having a FPGA program to deal with as well.
The specific micro I chose was the T89C51RD2 from Atmel. This is a 8052 clone with a couple of interesting features. It can be programmed directly via the serial port using standard Intel hex files and also has 64k of EEPROM program space, nifty for holding the FPGA fuse bit file, more on this later.
If you have done embedded Ethernet work before you will know that there is a limited range of Ethernet controller chips suitable for direct interfacing to embedded systems. The most popular pair are the Realtek 8019 and the Cirrus 8900. The choice is also limited for 100Mbps controllers. Here we have a more complex choice, as the standard design has a physical split between the digital management functions (MAC) and the analogue physical interface (PHY). Sometimes these functions are implemented a two separate interconnected chips, or sometimes implemented into one single chip. The interface between these two functions is a standard bus denoted MII. The PHY side handles all the analogue functions, buffering amplifying, collision detection etc, and presents the data to the digital chip as four parallel bits clocked at 25MHz. The MAC side of the pair is purely digital, and consists of an "engine" that implements the relevant parts of the 802 stack protocol functions in firmware. The resulting Ethernet frames are stored in DMA memory within the chip, ready for collection by the micro via the bus interface.
Fig 3 Caption: Inside the LAN Ethernet controller. The smsc 91C111 performs both MAC and PHY functions within the one package. It also contains separate paths for 10Mbps and 100Mbps operation. It can interface to micros via a 32,16 or 8 bit interfaces.
A rather convenient chip to use is the SMS LAN91C111. FIG3 shows a simplified layout of its internals. The chip combines 10Mpbs and 100Mbps transceivers and offers full MAC functionality with PHY line management, all in the one chip. The MII interface is also brought to the outside world, so only half the chip can be used if required. For example, when used with a separate optical PHY interface module. This chip supports 10BaseT and 100BaseTX half or full duplex, either switchable, or selectable via an internal auto sense mode.
The host interface to the controller is via a standard I/O mapped space arranged as 16 registers over three banks. Banks are selected in the normal way via a byte write to one of the registers, which is mirrored into all the banks. Data interface with the host is either 32, 16 or even 8 bit format. There is an internal FIFO buffer, which can be allocated dynamically on the fly for payload transmit or receive. A hardware DMA interface facility is also available for very fast I/O transfers. However, I found in practice that standard I/O transfers at 16 bits managed full 100Mbs throughput quite happily.
Reading and writing data to transmit and receive buffers is very straightforward. For receive, just sense a pin in the receive status register (which can also be arranged to generate an interrupt). To read the DMA data, we allocate a pointer, and read consecutive bytes/words as required. The first two words in the buffer are an error status and size of the data available in bytes.
To transmit data we first need to allocate a memory buffer. This can take some time, so the transmit function will wait for a short time in a loop. The data block must be preceded with two 16 bit words, a status word, and the number of bytes to be sent. Even if we don't know how many bytes we are going to transmit at this stage, we can always reset the write pointer and fill this location just before actual transmission. Following this operation we can now input the payload data sequentially into the DMA register. When finished, we command the MMU to queue the data and transmit to line. Lastly, we release the memory allocated.
All pretty simple stuff really, only that we need to transfer our data at intervals of 80 nanoseconds or so per write to keep up with the overall 100Mbps rate. Apart from this, the SMSC device is remarkably similar to its Realtek or Cirrus 10Mbps counterparts.
So much for the overview, the SMSC data sheet and the various application notes contain pseudo code listings for initialisation and for basic data transfer procedures, so I shall not delve on these matters here.
Ah, but I didn't quite tell the full story. Housekeeping functions in 100Mbps devices are rather more complex than those in their simple 10Mbps cousins. This is reflected in the complexity of the options required for setting up initial registers. For example the chip can be set for 10Mbps, 100Mbps, or an auto-negotiate mode (where the chip negotiates with the hub for best speed). Half and full duplex modes are also available. Oh yes, the negotiation sequences at power on can take some time, and may not always succeed (i.e. the hub may not support the required functions), so we need to take care of this as well.
And there is more. Initialising the chip is done by writing bytes into a handful of I/O mapped registers in the MAC section. The PHY section of the chip also needs to be initialised, but guess what, there is no direct register access to the PHY section. How do SMSC do it? Via the MII interface of course, and by bit blatting a serial stream via two dedicated lines. Where do we come in? Yes, we have to write the software in the host to bit blat this serial stream via bits in one of the MAC registers. This is not such a difficult thing to do, but it will add a few more hours under the hammer. Fortunately, examples are provided in the application notes.
It is a bit annoying that that this is not very clearly explained in the data sheets. Another point of confusion is that, in order to initialise certain functions, similar sounding register names have to be set in both the MAC and the PHY sections. Again, not very clearly documented and can cause a bit of frustration, it did to me anyway.
For this project, I used Altera's Acex family EP1K50 , for no other reason that it was available at the time, and that I have used it in a number of projects before. One of the useful advantages of this particular device is that its I/O pins are 5 volt tolerant when the chip is run from a 3.3 volt supply. In a new design I would be looking at a more up to date device such as Cyclone. For FPGA development I used the Quartus II development environment, freely available from the Altera website (also available as a CD ROM if you ask them nicely).
The FPGA talks to the micro via I/O parallel data pins, making use of the ALE, RD and WR lines in old standard 8051 access mode. In this context, the FPGA looks to the micro just as a standard outboard peripheral. The FPGA appears to the software as a collection of read write registers, some read only, some write only. Other registers are directly mapped to the FPGA's internal RAM. The FPGA RAM is used to store the core CPU program and for storing input and output payload data.
If you have never used FPGAs before, it is not all that difficult to get started. The learning curve is steep but worth the while. You don't even need to spend vast amounts of money, the circuit in Fig 4 and the free downloadable IDE from the manufacturer's is all you need (and plenty of spare time too). The simplest way to get started is to download the IDE and follow the built-in tutorial. You don't even need the programmer or any hardware while you are learning. results are simulated and displayed on your PC screen as waveforms.
Fig4 caption: Simple FPGA programmer. This works with most Altera devices. It is connected via the PC parallel port and used in conjunction with the Quartus IDE.
Almost all FPGA's need start up booting. This means transferring a "fuse file" from an external permanent memory such as EEPROM into the FPGA at power on. Even though specialist "configuration" devices are available, I decided to use the microprocessor itself to store the fuse file and to provide the programming timing pulses. With the Atmel 8051 having 64k of code space, there should be plenty spare. The timing requirements and file formats are all well documented and they are all simple to implement, it also saves me from having to include an extra EEPROM device on the board and to provide an extra program pod interface. In other words, I don't need the circuit shown in Fig5 to program the FPGA, the micro will do that or me. Quartus can generate a binary image (.RBF) file, representing a bit for bit image of the mask to be loaded into the FPGA. The only thing I needed to do was to write a bit of PC code to convert this to a standard HEX file and download it to the 8051 somewhere near the top of the 64k memory space.
One thing I didn't realise at the time was that the RBF file generated by Quartus was about 98k bytes long, far more than the 64k memory space available in the 8051, bother!
Before heading for the bottle and summoning my friend the wizard, I did a quick visual check of the binary file. This revealed a lot of redundancy in the form of long strings of zero bits. Mustering my years of experience I figured that a simple run length compression algorithm should reduce the size of the file, at least to within the size of the 8051 code area. The encoding method has to be very simple as well in order to allow the micro to decode the file in fast enough time as it ditches the bits into the FPGA.
A crude statistical analysis showed that repeated strings of zeroes are very common, but repeated strings of other bytes are not. The encoding method I eventually used is very simple.. Strings of two to 255 zero bytes long are encoded as a two byte sequence <55hex><nnhex> where nn is the number of zeroes in the string. A single zero byte is not encoded as a byte pair, it is kept as a single byte as there are quite a few of these. Any occurrences of the byte 55hex in the file are encoded as the byte pair <55><00>. The escape byte 55hex was chosen as one that does not occur very often in the file.
A better alternative might have been to move up to Cyclone, with it's own built program compression, but at this late stage, I decided I would not be starting up again!
Oh gosh, here comes the boring bit. At power on, the micro dumps the uncompressed RBF bit file into the FPGA. After this, the micro performs a simple test by reading back one of the registers from the FPGA on the simple assumption that if it can read it, the FPGA must be working.
After the FPGA is loaded, the micro initialises various FPGA registers, including the CPU core program (more on this later). Next, the micro takes direct control of the Ethernet controller by switching the multiplexer over to its data pins. It then proceeds to initialise the Ethernet controller, its buffers, MAC addresses and mode selection. The Ethernet chip is then instructed to go into "speed auto negotiate" which can take a few seconds. If the chip cannot negotiate 100Mbps, or if it cannot find the hub, it notifies the software, which enters into a sulk mode.
After all the initialisation code, the micro goes into simple Ethernet handling. This includes receiving frames, handling ARP and ICMP ping maintenance.
During most of the initial development I didn't use the FPGA at all. All code, including data transfers and UDP connections were done initially on the micro as function placeholders. The FPGA simply acted as a data feed through device between the micro and the Ethernet controller chip. Leaving the FPGA out of the way during development made life a lot easier, I can tell you!
Luckily, there are very few differences between writing IP code for a 10Mbs system and a 100Mbps one. So the snippets of code I had already used from a previous Realtek incarnation came in pretty handy here. The SMSC chip only required a few changes to the low level drivers, but that was it.
Having been satisfied that it all worked properly, it was time to start the FPGA turbocharger. In normal operation, the FPGA is brought into play when a block of payload data is available for transfer. The 8051 passes control to the FPGA only at these points.
Datagram transmission is initiated when data in the input FIFO within the FPGA has reached a certain size. In this project, an external A/D converter feeds continuous data into the FIFO via its own parallel port. When full, a flag is set which starts a program cycle. This causes the FPGA CPU core to take over the Ethernet controller chip and dump the contents of the FIFO directly into the Ethernet chip's own DMA area.
In practice, the FPGA hardware dumps three separate RAM buffers, one containing the Ethernet Header, another containing the IP and UDP headers and a third (the FIFO) containing the payload (the A/D data). The first two RAM areas are preloaded by the micro during call set-up and contain relatively static information such as MAC and IP addresses (source and destination). I could have chosen to let the micro the dump the Ethernet and IP/UDP headers from its own internal program, and them pass control to the FPGA only when the actual FIFO payload needs to be transmitted. Well, maybe next time.
A similar procedure is used to receive data, only in reverse.
Did I say I was going to implement a CPU core within the FPGA? Isn't that a bit over the top?
To keep things in perspective, the FPGA data transfer processor needs to do quite a few things. It has to wait until it receives a notification from the FIFO that it is full, then run a fixed sequence to modify some registers, calculate checksums and dump the words into the Ethernet controller. When its done, it all goes back into limbo mode, until the next request is received. The data receiver works in the same way but in reverse.
A programmer would feel at home writing computer code to perform these tasks, however, in hardware, things are slightly different.
One option is to use state transition table machines. During each state, the registers and logic perform one function. At the next state a different function and so on. When the complexity reaches a given size, it is actually better to implement these things using a CPU core. After all, a CPU is nothing but a collection of gates and registers running a state table, so why not use part of the FPGA to emulate a known CPU structure bit for bit?
I soon realised I was going to need some custom instructions, e.g. a one’s complement adder to perform internet checksums, so I decided against using a ready made core, such as NIOS, and decided to implement my own, thus satisfying every engineer's boyhood dream of inventing your own CPU. Fig 5 shows the results.
Fig 5 Caption: Overview of a simple custom CPU core. The need for specialist instructions, and the simplicity of the operation can make a customised core a better design option than a ready made off the shelf one.
Being aware of the wizard's warning about re-inventing the wheel I realised that the resultant CPU must be simple and straightforward to design and more importantly, to program. At the same time, I was aware it would possibly take me more time to get conversant with an existing off the peg CPU core, which possibly would not run fast enough.
Some basic decisions needed to be made for designing the core. I reached for my copy of "Dummies guide for designing your own CPU" to get some ideas. To make life easy all instructions are to run in a single machine cycle. This can be greatly helped by making the instruction size as wide as possible. I found that an 18 bit wide instruction set was about right for this job. Placing instructions and operands in separate bit slots creates an orthogonal instruction set that is easy to synthesize and visualise, albeit inefficient in storage, but this didn't matter too much here.
The application program required was quite small, and I decided to allocate 256 words of FPGA dual port RAM for CPU program memory. These allow for up to 256 instructions of 18 bits each, plenty for this application. This program memory area is written to by the micro once during power on. They are accessed as read only by the CPU core instruction counter. Very simple so far.
The cpu cycle times are dictated by the read and write setup times of the Ethernet Controller chip. The specs say it requires a minimum of 80nS for reads and writes, with a 100nS "rest" time in between. The Controller also provides a signal (RDYON), which is used to halt the CPU clocking until the operation is performed.
The FPGA is clocked at 80MHz. Using a 16 clock cycle results in the timing sequence shown in FIG 6
Fig 6 Caption Timing diagram and basic orthogonal instruction set for the core CPU. This is a vanilla basic design with all operations performed within one instruction cycle, and using a plain orthogonal 18 bit wide instruction set for simplicity
The main cycle time signal occurs once per cycle, and is used to increment the program counter and to clock enable all the internal registers, thus ensuring all the registers are clocked at the same time. An exception is the write pulse to the Ethernet controller, which needs to be somewhat narrower to meet the specs. A wider version of the write pulse is used to manage the data multiplexer between the 8051 and the Ethernet chip, this is shown as nENA in the diagram .
After an instruction clock edge, and at the end of cycle 0, the instruction will have been fetched, all the registers will have placed their value on the bus, and the result of any operation will be present to the registers. The multiplexer will take some time to settle (because of the slow bus interface) before reading or writing to the Ethernet controller. The wait signal from the Ethernet controller (IORDY) is connected to the enable input of the cycle counter. When and if the controller is waiting, the cycle counter will stop until the Ethernet controller is ready.
There is one direct interface to the 8051 via the general purpose 8 bit register REGA. One of the bits (bit7) is used stop reset the CPU core program. When the 8051 sets this bit, the CPU core starts from program address zero. The reset is performed asynchronously, which is perfectly harmless in this context.
I resisted all temptation of designing a full set of instructions and stuck only to those necessary to run the program. As can be seen in Fig5 there are two I/O registers the core can read, three RAM data areas and three 16 bit general purpose registers. The arithmetic unit performs some of the basic operations such as add, move and copy data to and from the Ethernet controller. There are also a number of specialist instructions. It is these specialist instructions that gives the core its speed and performance. For example, one of the instructions performs all these operations in parallel:
Copy a block of 16 bit words from the RAM area to the Ethernet controller Data register. Register A contains the starting address in the RAM, Register B contains the number of words to copy. An Internet checksum is calculated on the fly as the transfers are made, and accrued into register C.
A nifty way of calculating the internet one's complement checksum is by using a full adder and feeding the carry output back to to the carry input.
When the transmit (or receive) program is kick started by the 8051, it runs the relevant set of instructions until it reaches the end and then it stops. During this time it will have copied the contents of RAM to the Ethernet controller, and updated the IP datagram variable fields such as checksums, ID counter and others as necessary. The Ethernet controller is then triggered for transmission and the cycle will repeat as soon as there is more data to transmit.
Fig 7 Caption: Sequence of headers and payload data as queued for transmission. Some fields are constant for the duration of the session, some others, such as the internet checksum, need to be calculated on the fly by the core CPU within the FPGA.
FIG7 shows the three data block headers that the controller needs for a typical UDP datagram transmission. The first block is the Ethernet Frame Header, consisting of source and destination MAC addresses plus a protocol field. This data does not change during a session, and is transmitted without change for every block transmitted.
The next two headers, the IP and UDP headers are a bit more complicated. These require some fields, such as checksum to change for every frame transmitted, how is this handled?
Looking at the figure provides some answers. The 8051 handles call set up, that is the original ARP requests and any information requested by the remote. This allows the micro to load session dependent variables such as remote MAC, IP and port addresses into the various unchangeable fields in the corresponding headers. These fields will not change during a session, and for all intents and purposes all the information in the three headers are constants for the session. To simplify things, all UDP datagrams transmitted are of the same size, no fragmentation is provided, and no option fields is included within the IP header. This will also make some other fields unchangeable.
At this point, there are only three fields that need dynamic updating: The IP header ID number, the IP header checksum, and the UDP header checksum.
The general rule is that the IP header's ID number has to change for every datagram transmitted. In practice, it is incremented modulo 2^16. This also means that the IP header's checksum has to be recalculated. This is because the IP header checksum is the sum of all the fields in the IP header. Fortunately, there is no need to recalculate the IP header checksum every time. Its value can be simply decremented by one as the ID field is incremented by one, thus maintaining a constant sum. Needles to say, a special instruction in our CPU core is used specifically to do this.
The UDP checksum is somewhat more complicated, as it needs to be calculated over the payload data as well. It also needs to be written into the array before the payload is entered. As our payload data arrives in real time, this is a rather difficult thing to do. One option would be to use no UDP checksums at all, as this is perfectly allowed in the UDP specifications, but we are going to be a bit cleverer than that. Remembering that I am in control of the payload formats used, I decided to add an extra 16 bit word at the end of the payload field. This is to be ignored by the receiver end. But it allows me to place a dummy checksum at this point. The original checksum entry in the UDP header is just filled with a constant value, and the last word is filled with the accrued checksum as calculated by the CPU engine. The important thing here is that all the words in the datagram must add to 0xFFFF. So it doesn't really matter where the checksum is placed.
I decided to download the CPU core instruction set to the FPGA from the micro during power on rather than hardwire the instruction set during FPGA compilation. As the CPU core program is fixed, either method would be equally suitable. The reasoning behind this was rather simple: time saving during development. A typical FPGA compilation-program load cycle can take several minutes, whereas a compile run on Keil's IDE just a few seconds. The code is declared within the C compiler source as a list of constant 16 bit data values. The CPU core program is stored in the 8051's program ROM space. When the 8051 fires, it copies this to the FPGA's program RAM.
In order to provide some sense of proportion, I decided to implement a rather crude CPU core "assembler" using C's define instructions. A very crude method, but it works. It also avoids me having to spend the rest of the season writing an assembly translator for this CPU.
The box shows the results. I have shown only a few of the instructions to show the principle. The orthogonal instruction set ensures each part of an instruction can be assembled separately as an "OR" combination of #define statements. One set of #defines is the list of opcode, the next is the list of registers, the next forms either a constant, or a source/destination combination.
|
Working with mixed development technologies can be a nightmare. At any one stage of the development, the PC screen would be showing the Microsoft MSC IDE, the Keil microcontroller IDE, the Quartus IDE and the Acrobat reader. At times, small changes to the design required a domino sequence of recompilations for each of the IDEs in turn. These were rather lengthy operations, which resulted in me drinking far more coffee and chewing more sweets than I should have liked. Nevertheless it was very reassuring to see a complex design coming out of the box that actually worked.
Eddie Insam lives near the Thames in southern England. He has over 20 years of experience working in innovative telecommunications and signal processing designs. Specialising in audio and image processing, he has written several articles and a book on a number of related subjects. He wastes the rest of his time trying to stay in a straight line rowing boats You can reach him at edinsam@eix.co.uk and visit his website at www.eix.co.uk.
www.Smsc.com Various data sheets and application notes on the SMSC-LAN91C111 Ethernet Controller Chip
Book: "TCP/IP Embedded Internet Applications". Eddie Insam. Newnes Publications, 2003. www.newnespress.com , ISBN 0750657359
www.Altera.com Datasheets, information on FPGA devices and downloadable development IDE.