Hello,

I'm trying out a PCIe demonstration design for Xilinx devices. I'll confugure the device on a development board and plug it into an open PCIe slot on a PC running Fedora. I was reading over the documentation, and in the summary of the demo design chapter, it says this:

Users can leverage standard operating system
utilities that enable generation of read and write transactions to the target space in the​

reference design.

This is on page 263 of this document: http://www.xilinx.com/support/documentation/ip_documentation/pcie_7x/v1_7/pg054-7series-pcie.pdf​

I have no earthly idea what those utilities might be. Has anyone any ideas?​

 

Hello,

I'm trying out a PCIe demonstration design for Xilinx devices. I'll confugure the device on a development board and plug it into an open PCIe slot on a PC running Fedora. I was reading over the documentation, and in the summary of the demo design chapter, it says this:

This is on page 263 of this document: http://www.xilinx.com/support/documentation/ip_documentation/pcie_7x/v1_7/pg054-7series-pcie.pdf​

I have no earthly idea what those utilities might be. Has anyone any ideas?​

Page 397 talks about using LSPCI...this should let you see the configuration of the device, if I read that properly..

 

Use lspci with the -v command.

You will be able to see what your device is in the list, even though there isn't a driver attached to it. If there is a driver attached to it, it will be listed.

If you don't have the interface driver for it, you'll need to write one as a loadable module.

lspci -v

02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 02)
        Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
        Flags: bus master, fast devsel, latency 0, IRQ 1278
        I/O ports at de00 
        Memory at fdfff000 (64-bit, prefetchable) [size=4K]
        Memory at fdfe0000 (64-bit, prefetchable) [size=64K]
        [virtual] Expansion ROM at fdf00000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/2 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable- Count=2 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [160] Device Serial Number 12-34-56-78-12-34-56-78
        Kernel driver in use: r8169
        Kernel modules: r8169

03:07.0 Ethernet controller: 3Com Corporation 3c905B 100BaseTX [Cyclone] (rev 64)
        Subsystem: 3Com Corporation 3C905B Fast Etherlink XL 10/100
        Flags: bus master, medium devsel, latency 32, IRQ 21
        I/O ports at cf00 
        Memory at fdcff000 (32-bit, non-prefetchable) 
        [virtual] Expansion ROM at fdb00000 [disabled] [size=128K]
        Capabilities: [dc] Power Management version 1
        Kernel driver in use: 3c59x
        Kernel modules: 3c59x

 

If you don't have the interface driver for it, you'll need to write one as a loadable module.

I am so not a driver writer. We will have a driver for the target platform, but what I'm trying to do is just get a baseline system running on a PC for now. All I want is something to write and read to the attached memory of the example design, but I don't have the resources to create a driver. (end of sob story)

 

If you are using a PCIe IP block, it should have come with code and a makefile to at least see it on the bus from the CPU's point of view.

If you wrote the PCIe block rather than using a complete IP module, I'd suggest trying a complete IP module, though they can get into "ludicrous range" for pricing.

 

If you are using a PCIe IP block, it should have come with code and a makefile to at least see it on the bus from the CPU's point of view.

It should. But it doesen't. My development kit is supplied with a software driver, but I am using a different design than the one delivered with the kit. I'm not sure the driver will work, but I will try it when I get the lab computer I've requested.

 

I found this:

A simple user app can be written to communicate with the PCIe endpoint.

Note that you don’t have to write a driver for simple interactions with the endpoint.


The pcie resource is a file that represents a bar on the pcie endpoint. resourceN where N = 0 through 6 for the number of bars you have configured. To find the resource file do the following.

From the linux command prompt execute the following command

“Lspci –vv”

This command reports the pci and pcie configuration for the host system.

Look in the output for
01:00.0 Memory controller: Xilinx Corporation Device 0505

Note the numbers to the left. This number represents the bus and card address. This will be a unique number for your system.

Now using your bus and card address number we can locate the resource file on your linux system.

Navigate to the following directory

/sys/bus/pci/devices/0000:01:00.0

Substitute your bus and card address for the right most folder.

Inside this folder you should see resource0. This is bar0 in your pcie endpoint block.

Now you can write an app to access your pcie card.

User space program layout

Open pcie resource
Mmap pcie resource
Perform io operations
Read
Write
Unmmap pcie resource
Close pcie resource

Once you mmap the resource you will be able to read and write to the bar as though it is a variable.

Note that you need to have root privileges to open and mmap the resource file.

Interesting. I'm not sure what it all means, but I'll figure it out.

 

The format of the output of lscpi is:

[[[[<domain>]:]<bus>]:][<slot>][.[<func>]]

So you are seeing in the output pcibus:slot.function in the example I posted above.

 

I use that information to find the resource in the directory hierarchy, then open the resource for basic reading and writing. I got everything except for the "Mmap" and "Unmap" but I'm sure I will find the info easily by gurggling it.

 

Memory map your functions on the board, then free the system memory once the application is complete.

 

Ah! So memory map creates an entry in system memory space for the device!? Ok, that makes sense.

 

Memory mapped functions are the simplest to implement.

In the end, you have a function and it is mapped to an address, and the output of that function is at another address. So you write to one address, read the result on another address. The OS and PCIe IP on the board handle the actual bus transfer. If the result is an output from the card, then that should change states, and no read is required. It depends on how you set up your hardware, and which IP block you are using.

 

I'm still not getting the mmap stuff. Why couldn't I just do this:

>lspci -vv
>01:00.0 Memory controller: Xilinx Corporation Device 0505

#include <studio.h>
#include <stdlib.h>
int main() {
FILE *dev;
int data;
if((dev = open("/sys/bus/pci/devices/0000:01:00.0", "wb") == NULL) {
printf("Failed to open device\n");
return 1;
}
write(dev, 0xFF);
close(dev);
}

 

Ok, here's what I tried:

#include <stdio.h>
#include <stdlib.h>
int main() {
FILE * dev;
int data;
if(dev = open("sys/bus/pci/devices/0000:02:00.0", "w+b") == NULL) {
printf("Failed to open device\n");
return 1;
}
write(dev, 0xaa);
printf("Data written to device 0xaa\n");
data = read(dev);
printf("Program Terminating with data %x\n", data);
close(dev);
return 0;
}

Where lspci returned: 02:00.0 Xilinx...

The program ran through, but shows the read data = 0 in the last printf() statement.

 

On the FPGA, configure the PCIe block to simply loop any input back to the output.

See if that gives a response.

Then work from there.

 

PCIe transaction model doesn't allow that. Reads are split transactions, initiator sends read request, completer sends results. I am required to buffer the data to be read back.

BTW, I fixed the typo in the path in the above code, but stil doens't work. I'm getting to give up on this and try something different.

 

I talked to a real software engineer, and he told me that mmap is absolutely required. I had omitted it because I still don't understand it. Guess I better wrap my head around it if I want to make this work.

Does anyone have an example of mmap'ing a device?

 

Ok, a little progress. First of all, the line:

if((dev = open("/sys/bus/pci/devices/0000:01:00.0", "wb") == NULL)

was wrong. "open" returns -1 when it fails, and this was failing, but I didn't know it. Once I corrected the line, I got an indication that the device failed to open. After awhile, I realized that I needed root priviliges to run this program, due to tyring to opoen a device. The device opened, but the program exited when I tried to read to it.

So, considering what the software engineer told me, I replaced the read with this:

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/mman.h>
 
int main() {
FILE * dev;
int rdata = 0;
int wdata = 0xaa;
int objs;
int *map;
#define FILESIZE (1 * sizeof(int))
 
if(dev = open("/sys/bus/pci/devices/0000:02:00.0/resource0", O_RDWR | O_CREAT | O_TRUNC, (mode_t) 0600) == -1) {
printf("Failed to open device\n");
return 1;
}
 
map = mmap(0, FILESIZE, PROT_READ | PROT_WRITE, MAP_SHARED, dev, 0);
if(map == MAP_FAILED) {
printf("Error mapping device\n");
return 1;
}
 
if(munmap(map, FILESIZE) == -1) {
printf("Error un-mapping the file\n");
}
 
close(dev);
return 0;
}

But now I get the "Map Failed" when I run. Could it be that the OS won't allow me to map a device?

 

Please see the above post again. This time, I have all my code posted. Same error.

 

I think I've narrowed down the problem with mmap. When I open a device, Linux treats it as a file with size = 0. The mmap function fails if the size of mapping requested (the macro "FILESIZE" in this case) is larger than the file(device) being mapped. I don't now how to fix this, but I might have another method.

Still, I'm open to any ideas about how to fix the file size. FYI, I tried to map with zero specified for size to match the device, but no dice.