Using UHD 4.6/Ubuntu 22.04/x310, I have built many images in the last year or so with rfnoc_image_builder. Recently in the last month, I get the following error on almost all images: [image: image.png] I have tried the following, after referencing this the known issues section in the USRP3 build instructions ( https://files.ettus.com/manual/md_usrp3_build_instructions.html): 1. doing the suggested and making a non-functional source code change and recommitting the git 2. deleting the .git directory in both the block directory and the uhd/ directory where the fpga build happens 3. changing the build seed in uhd/fpga/usrp3/top/x300/Makefile 4. Running on a different machine, copying the block source code and using a different UHD git all together (private rehost vs the github UHD). The vivado 2021.1 install is the same as its on a network file system These do not produce repeatable good results. Maybe once a week or once every two weeks one of these things will finish the build. This has been happening for about a month or two, and I don't know how else to troubleshoot. Any advice? Thanks, David

1.  doing the suggested and making a non-functional source code change
and recommitting the git
2. deleting the .git directory in both the block directory and the
uhd/ directory where the fpga build happens
3. changing the build seed in uhd/fpga/usrp3/top/x300/Makefile
4. Running on a different machine, copying the block source code and
using a different UHD git all together (private rehost vs the github UHD).
The vivado 2021.1 install is the same as its on a network file system

Hi David, I'm surprised that you're seeing it that frequently. The Ettus continuous integration tests build FPGAs regularly and from what I understand this issue is pretty rare there. This makes me wonder if there's something about the images you're building that causes this to reproduce more frequently for you. Can you estimate what percentage of unique builds (with a unique build seed or git hash) fail? Which FPGA image are you building? Does it have custom logic in it? You could use the repeat_fpga_build.py script to automate building the FPGA multiple times to get a successful build. It automates the process of selecting a unique seed for each build and can even run multiple build jobs at a time. Thanks, Wade On Mon, Mar 3, 2025 at 1:30 PM David <vitishlsfan21@gmail.com> wrote: > Using UHD 4.6/Ubuntu 22.04/x310, I have built many images in the last year > or so with rfnoc_image_builder. Recently in the last month, I get the > following error on almost all images: > > [image: image.png] > > I have tried the following, after referencing this the known issues > section in the USRP3 build instructions ( > https://files.ettus.com/manual/md_usrp3_build_instructions.html): > > 1. doing the suggested and making a non-functional source code change > and recommitting the git > 2. deleting the .git directory in both the block directory and the > uhd/ directory where the fpga build happens > 3. changing the build seed in uhd/fpga/usrp3/top/x300/Makefile > 4. Running on a different machine, copying the block source code and > using a different UHD git all together (private rehost vs the github UHD). > The vivado 2021.1 install is the same as its on a network file system > > These do not produce repeatable good results. Maybe once a week or once > every two weeks one of these things will finish the build. This has been > happening for about a month or two, and I don't know how else to > troubleshoot. > > Any advice? > > Thanks, > > David > _______________________________________________ > USRP-users mailing list -- usrp-users@lists.ettus.com > To unsubscribe send an email to usrp-users-leave@lists.ettus.com >

1.  doing the suggested and making a non-functional source code
change and recommitting the git
2. deleting the .git directory in both the block directory and the
uhd/ directory where the fpga build happens
3. changing the build seed in uhd/fpga/usrp3/top/x300/Makefile
4. Running on a different machine, copying the block source code and
using a different UHD git all together (private rehost vs the github UHD).
The vivado 2021.1 install is the same as its on a network file system

Wade, Thank you for this, I took a look at repeat_fpga_build.py and I think that will be very helpful. I am going to add some modifications that log the types of failures I get and the amount. I have the issue with two different builds, both with two custom RFNoC blocks and HLS modules for my logic. I am building an x310 image, and both of these builds have completed before with the same HLS verilog files used. One thing that is worth mentioning is I get a timing violation in a pulse stretcher module from UHD. However, when I have gotten that error the build generates the bit file and it still works for my use case on the FPGA. That has been around for some time and I don't think it is the cause of this recent issue. I will try and get the stats from the repeat_fpga_build.py and reply in the coming days. Thanks, David On Thu, Mar 6, 2025 at 9:01 AM Wade Fife <wade.fife@ettus.com> wrote: > Hi David, > > I'm surprised that you're seeing it that frequently. The Ettus continuous > integration tests build FPGAs regularly and from what I understand this > issue is pretty rare there. This makes me wonder if there's something about > the images you're building that causes this to reproduce more frequently > for you. Can you estimate what percentage of unique builds (with a unique > build seed or git hash) fail? Which FPGA image are you building? Does it > have custom logic in it? > > You could use the repeat_fpga_build.py script to automate building the > FPGA multiple times to get a successful build. It automates the process of > selecting a unique seed for each build and can even run multiple build jobs > at a time. > > Thanks, > > Wade > > On Mon, Mar 3, 2025 at 1:30 PM David <vitishlsfan21@gmail.com> wrote: > >> Using UHD 4.6/Ubuntu 22.04/x310, I have built many images in the last >> year or so with rfnoc_image_builder. Recently in the last month, I get the >> following error on almost all images: >> >> [image: image.png] >> >> I have tried the following, after referencing this the known issues >> section in the USRP3 build instructions ( >> https://files.ettus.com/manual/md_usrp3_build_instructions.html): >> >> 1. doing the suggested and making a non-functional source code >> change and recommitting the git >> 2. deleting the .git directory in both the block directory and the >> uhd/ directory where the fpga build happens >> 3. changing the build seed in uhd/fpga/usrp3/top/x300/Makefile >> 4. Running on a different machine, copying the block source code and >> using a different UHD git all together (private rehost vs the github UHD). >> The vivado 2021.1 install is the same as its on a network file system >> >> These do not produce repeatable good results. Maybe once a week or once >> every two weeks one of these things will finish the build. This has been >> happening for about a month or two, and I don't know how else to >> troubleshoot. >> >> Any advice? >> >> Thanks, >> >> David >> _______________________________________________ >> USRP-users mailing list -- usrp-users@lists.ettus.com >> To unsubscribe send an email to usrp-users-leave@lists.ettus.com >> >

Hi David & Wade, I would just like to chime in on this that I've been having exactly the same issue, repeatably, building for N310&320 series devices. I'm running a heavily customised image with plenty of (known good) custom logic, and the error comes up seemingly at random following a commit. The only (semi-) repeatable way I've found of fixing this is renaming the image_core.yml file from which rfnoc_image_builder is running. Strange, I know, though it seems to work 50% of the time, at least for me. If you find any tricks other than those previously mentioned please let me know. I'm going to do some digging once I'm not under deadline-pressure, as this issue is getting on my nerves somewhat. Kind Regards, Sam ________________________________ From: Wade Fife <wade.fife@ettus.com> Sent: 06 March 2025 17:01 To: David <vitishlsfan21@gmail.com> Cc: USRP-users@lists.ettus.com <usrp-users@lists.ettus.com> Subject: [USRP-users] Re: rfnoc_image_builder error: trying to tool lock on already tool locked arc Hi David, I'm surprised that you're seeing it that frequently. The Ettus continuous integration tests build FPGAs regularly and from what I understand this issue is pretty rare there. This makes me wonder if there's something about the images you're building that causes this to reproduce more frequently for you. Can you estimate what percentage of unique builds (with a unique build seed or git hash) fail? Which FPGA image are you building? Does it have custom logic in it? You could use the repeat_fpga_build.py script to automate building the FPGA multiple times to get a successful build. It automates the process of selecting a unique seed for each build and can even run multiple build jobs at a time. Thanks, Wade On Mon, Mar 3, 2025 at 1:30 PM David <vitishlsfan21@gmail.com<mailto:vitishlsfan21@gmail.com>> wrote: Using UHD 4.6/Ubuntu 22.04/x310, I have built many images in the last year or so with rfnoc_image_builder. Recently in the last month, I get the following error on almost all images: [image.png] I have tried the following, after referencing this the known issues section in the USRP3 build instructions (https://files.ettus.com/manual/md_usrp3_build_instructions.html): 1. doing the suggested and making a non-functional source code change and recommitting the git 2. deleting the .git directory in both the block directory and the uhd/ directory where the fpga build happens 3. changing the build seed in uhd/fpga/usrp3/top/x300/Makefile 4. Running on a different machine, copying the block source code and using a different UHD git all together (private rehost vs the github UHD). The vivado 2021.1 install is the same as its on a network file system These do not produce repeatable good results. Maybe once a week or once every two weeks one of these things will finish the build. This has been happening for about a month or two, and I don't know how else to troubleshoot. Any advice? Thanks, David _______________________________________________ USRP-users mailing list -- usrp-users@lists.ettus.com<mailto:usrp-users@lists.ettus.com> To unsubscribe send an email to usrp-users-leave@lists.ettus.com<mailto:usrp-users-leave@lists.ettus.com>

1.  doing the suggested and making a non-functional source code change
and recommitting the git
2. deleting the .git directory in both the block directory and the
uhd/ directory where the fpga build happens
3. changing the build seed in uhd/fpga/usrp3/top/x300/Makefile
4. Running on a different machine, copying the block source code and
using a different UHD git all together (private rehost vs the github UHD).
The vivado 2021.1 install is the same as its on a network file system

Wade and Sam, The repeat FPGA build script was very useful. I was able to let it run and find a build solution. That will let me proceed with my project, but I still want to nail down why this showed up. Some additional info: my design does not meet timing constraints, but it still builds the bit file. This has been the case for some time before this tool lock error came up. The bit file still works for my use case, so a failure for timing is still a success to me. The error in the post_route_timing_summary.rpt is: *Slack (VIOLATED) : -0.948ns (required time - arrival time)* * Source: x300_core/bus_int_i/rfnoc_sandbox_i/b_X0_6/noc_shell_X_i/pulse_stretch_min_ce/state_reg_replica/C* * (rising edge-triggered cell FDRE clocked by ce_clk {rise@0.000ns fall@2.333ns period=4.667ns})* * Destination: x300_core/bus_int_i/rfnoc_sandbox_i/b_X0_6/noc_shell_X_i/ctrlport_endpoint_i/gen_async_fifos.out_fifo_i/o_rst_sync_i/synchronizer_false_path/stages[0].value_reg[0][0]/D* * (rising edge-triggered cell FDRE clocked by bus_clk_div2 {rise@0.000ns fall@5.333ns period=10.667ns})* * Path Group: bus_clk_div2* The timing constraint slack is happening in my X block noc shell, most certainly because of what I am doing in my verilog design. However, because the bit file "works" as far as the output I am expecting, I made a design decision to continue on while not making the constraint. All of my useful work is done in IQ, and the noc shell slack seems to be on the ctrlport. I modified the script to continue on success and record success/failures to a csv. I am getting 19 failures and 11 successful builds over the 30 runs I did. [image: image.png] My next set of 30 runs will not change the build seed to get more data. Thanks, David On Thu, Mar 13, 2025 at 10:51 AM Sam Lane <sam.lane@surrey.ac.uk> wrote: > Hi David & Wade, > > I would just like to chime in on this that I've been having exactly the > same issue, repeatably, building for N310&320 series devices. I'm running a > heavily customised image with plenty of (known good) custom logic, and the > error comes up seemingly at random following a commit. > > The only (semi-) repeatable way I've found of fixing this is renaming the > image_core.yml file from which rfnoc_image_builder is running. Strange, I > know, though it seems to work 50% of the time, at least for me. If you find > any tricks other than those previously mentioned please let me know. > > I'm going to do some digging once I'm not under deadline-pressure, as this > issue is getting on my nerves somewhat. > > Kind Regards, > Sam > > > ------------------------------ > *From:* Wade Fife <wade.fife@ettus.com> > *Sent:* 06 March 2025 17:01 > *To:* David <vitishlsfan21@gmail.com> > *Cc:* USRP-users@lists.ettus.com <usrp-users@lists.ettus.com> > *Subject:* [USRP-users] Re: rfnoc_image_builder error: trying to tool > lock on already tool locked arc > > Hi David, > > I'm surprised that you're seeing it that frequently. The Ettus continuous > integration tests build FPGAs regularly and from what I understand this > issue is pretty rare there. This makes me wonder if there's something about > the images you're building that causes this to reproduce more frequently > for you. Can you estimate what percentage of unique builds (with a unique > build seed or git hash) fail? Which FPGA image are you building? Does it > have custom logic in it? > > You could use the repeat_fpga_build.py script to automate building the > FPGA multiple times to get a successful build. It automates the process of > selecting a unique seed for each build and can even run multiple build jobs > at a time. > > Thanks, > > Wade > > On Mon, Mar 3, 2025 at 1:30 PM David <vitishlsfan21@gmail.com> wrote: > > Using UHD 4.6/Ubuntu 22.04/x310, I have built many images in the last year > or so with rfnoc_image_builder. Recently in the last month, I get the > following error on almost all images: > > [image: image.png] > > I have tried the following, after referencing this the known issues > section in the USRP3 build instructions ( > https://files.ettus.com/manual/md_usrp3_build_instructions.html): > > 1. doing the suggested and making a non-functional source code change > and recommitting the git > 2. deleting the .git directory in both the block directory and the > uhd/ directory where the fpga build happens > 3. changing the build seed in uhd/fpga/usrp3/top/x300/Makefile > 4. Running on a different machine, copying the block source code and > using a different UHD git all together (private rehost vs the github UHD). > The vivado 2021.1 install is the same as its on a network file system > > These do not produce repeatable good results. Maybe once a week or once > every two weeks one of these things will finish the build. This has been > happening for about a month or two, and I don't know how else to > troubleshoot. > > Any advice? > > Thanks, > > David > _______________________________________________ > USRP-users mailing list -- usrp-users@lists.ettus.com > To unsubscribe send an email to usrp-users-leave@lists.ettus.com > >

1.  doing the suggested and making a non-functional source code
change and recommitting the git
2. deleting the .git directory in both the block directory and the
uhd/ directory where the fpga build happens
3. changing the build seed in uhd/fpga/usrp3/top/x300/Makefile
4. Running on a different machine, copying the block source code and
using a different UHD git all together (private rehost vs the github UHD).
The vivado 2021.1 install is the same as its on a network file system

Thanks David, that is helpful. I suspect that this tool lock bug shows up more frequently when Vivado is struggling to meet timing. If you fix that timing issue then the problem might become less frequent. It looks like that path is on a reset going to a "synchronizer_false_path", which should be a false path (hence the name). A false path means that Vivado should not be doing timing analysis on it and can make it as long as it wants. There is a constraint here that should be setting that as a false path: https://github.com/EttusResearch/uhd/blob/UHD-4.6/fpga/usrp3/top/x300/timing.xdc#L599 set_false_path -to [get_pins -hierarchical -filter {NAME =~ */synchronizer_false_path/stages[0].value_reg[0][*]/D}] Did you by chance modify the timing constraints? Or the file Makefile.x300.inc where that constraint gets pulled in? Wade On Sat, Mar 15, 2025 at 12:48 PM David <vitishlsfan21@gmail.com> wrote: > Wade and Sam, > > The repeat FPGA build script was very useful. I was able to let it run and > find a build solution. That will let me proceed with my project, but I > still want to nail down why this showed up. > > Some additional info: my design does not meet timing constraints, but it > still builds the bit file. This has been the case for some time before this > tool lock error came up. The bit file still works for my use case, so a > failure for timing is still a success to me. The error in the > post_route_timing_summary.rpt is: > > *Slack (VIOLATED) : -0.948ns (required time - arrival time)* > * Source: > x300_core/bus_int_i/rfnoc_sandbox_i/b_X0_6/noc_shell_X_i/pulse_stretch_min_ce/state_reg_replica/C* > * (rising edge-triggered cell FDRE clocked by > ce_clk {rise@0.000ns fall@2.333ns period=4.667ns})* > * Destination: > x300_core/bus_int_i/rfnoc_sandbox_i/b_X0_6/noc_shell_X_i/ctrlport_endpoint_i/gen_async_fifos.out_fifo_i/o_rst_sync_i/synchronizer_false_path/stages[0].value_reg[0][0]/D* > * (rising edge-triggered cell FDRE clocked by > bus_clk_div2 {rise@0.000ns fall@5.333ns period=10.667ns})* > * Path Group: bus_clk_div2* > > The timing constraint slack is happening in my X block noc shell, most > certainly because of what I am doing in my verilog design. However, because > the bit file "works" as far as the output I am expecting, I made a design > decision to continue on while not making the constraint. All of my useful > work is done in IQ, and the noc shell slack seems to be on the ctrlport. > > I modified the script to continue on success and record success/failures > to a csv. I am getting 19 failures and 11 successful builds over the 30 > runs I did. > > [image: image.png] > > My next set of 30 runs will not change the build seed to get more data. > > Thanks, > > David > > > On Thu, Mar 13, 2025 at 10:51 AM Sam Lane <sam.lane@surrey.ac.uk> wrote: > >> Hi David & Wade, >> >> I would just like to chime in on this that I've been having exactly the >> same issue, repeatably, building for N310&320 series devices. I'm running a >> heavily customised image with plenty of (known good) custom logic, and the >> error comes up seemingly at random following a commit. >> >> The only (semi-) repeatable way I've found of fixing this is renaming the >> image_core.yml file from which rfnoc_image_builder is running. Strange, I >> know, though it seems to work 50% of the time, at least for me. If you find >> any tricks other than those previously mentioned please let me know. >> >> I'm going to do some digging once I'm not under deadline-pressure, as >> this issue is getting on my nerves somewhat. >> >> Kind Regards, >> Sam >> >> >> ------------------------------ >> *From:* Wade Fife <wade.fife@ettus.com> >> *Sent:* 06 March 2025 17:01 >> *To:* David <vitishlsfan21@gmail.com> >> *Cc:* USRP-users@lists.ettus.com <usrp-users@lists.ettus.com> >> *Subject:* [USRP-users] Re: rfnoc_image_builder error: trying to tool >> lock on already tool locked arc >> >> Hi David, >> >> I'm surprised that you're seeing it that frequently. The Ettus continuous >> integration tests build FPGAs regularly and from what I understand this >> issue is pretty rare there. This makes me wonder if there's something about >> the images you're building that causes this to reproduce more frequently >> for you. Can you estimate what percentage of unique builds (with a unique >> build seed or git hash) fail? Which FPGA image are you building? Does it >> have custom logic in it? >> >> You could use the repeat_fpga_build.py script to automate building the >> FPGA multiple times to get a successful build. It automates the process of >> selecting a unique seed for each build and can even run multiple build jobs >> at a time. >> >> Thanks, >> >> Wade >> >> On Mon, Mar 3, 2025 at 1:30 PM David <vitishlsfan21@gmail.com> wrote: >> >> Using UHD 4.6/Ubuntu 22.04/x310, I have built many images in the last >> year or so with rfnoc_image_builder. Recently in the last month, I get the >> following error on almost all images: >> >> [image: image.png] >> >> I have tried the following, after referencing this the known issues >> section in the USRP3 build instructions ( >> https://files.ettus.com/manual/md_usrp3_build_instructions.html): >> >> 1. doing the suggested and making a non-functional source code >> change and recommitting the git >> 2. deleting the .git directory in both the block directory and the >> uhd/ directory where the fpga build happens >> 3. changing the build seed in uhd/fpga/usrp3/top/x300/Makefile >> 4. Running on a different machine, copying the block source code and >> using a different UHD git all together (private rehost vs the github UHD). >> The vivado 2021.1 install is the same as its on a network file system >> >> These do not produce repeatable good results. Maybe once a week or once >> every two weeks one of these things will finish the build. This has been >> happening for about a month or two, and I don't know how else to >> troubleshoot. >> >> Any advice? >> >> Thanks, >> >> David >> _______________________________________________ >> USRP-users mailing list -- usrp-users@lists.ettus.com >> To unsubscribe send an email to usrp-users-leave@lists.ettus.com >> >>

1.  doing the suggested and making a non-functional source code
change and recommitting the git
2. deleting the .git directory in both the block directory and the
uhd/ directory where the fpga build happens
3. changing the build seed in uhd/fpga/usrp3/top/x300/Makefile
4. Running on a different machine, copying the block source code and
using a different UHD git all together (private rehost vs the github UHD).
The vivado 2021.1 install is the same as its on a network file system

Wade, Took a look at the timing.xdc you sent, and mine is distinctly different. It also contains custom block names I've used in the past that have nothing to do with the design. No modifications were made to this file on my end. Searching for the "## Asynchronous Paths" comment on line 597, my local timing.xdc looks like: [image: image.png] It appears to be manually listing all the blocks, instead of the convenient star notation I see in the linked timing.xdc line 599. Interestingly, the name of the block I am working on "X" does not appear anywhere, only old block names made a long time ago. I'm not sure why the old block names are in there, or why the format of the timing.xdc file looks different. I will make a backup and use the file on the remote UHD-4.6. Thanks, David On Sat, Mar 15, 2025 at 1:09 PM Wade Fife <wade.fife@ettus.com> wrote: > Thanks David, that is helpful. I suspect that this tool lock bug shows up > more frequently when Vivado is struggling to meet timing. If you fix that > timing issue then the problem might become less frequent. It looks like > that path is on a reset going to a "synchronizer_false_path", which should > be a false path (hence the name). A false path means that Vivado should not > be doing timing analysis on it and can make it as long as it wants. There > is a constraint here that should be setting that as a false path: > > > https://github.com/EttusResearch/uhd/blob/UHD-4.6/fpga/usrp3/top/x300/timing.xdc#L599 > > set_false_path -to [get_pins -hierarchical -filter {NAME =~ > */synchronizer_false_path/stages[0].value_reg[0][*]/D}] > > Did you by chance modify the timing constraints? Or the file > Makefile.x300.inc where that constraint gets pulled in? > > Wade > > On Sat, Mar 15, 2025 at 12:48 PM David <vitishlsfan21@gmail.com> wrote: > >> Wade and Sam, >> >> The repeat FPGA build script was very useful. I was able to let it run >> and find a build solution. That will let me proceed with my project, but I >> still want to nail down why this showed up. >> >> Some additional info: my design does not meet timing constraints, but it >> still builds the bit file. This has been the case for some time before this >> tool lock error came up. The bit file still works for my use case, so a >> failure for timing is still a success to me. The error in the >> post_route_timing_summary.rpt is: >> >> *Slack (VIOLATED) : -0.948ns (required time - arrival time)* >> * Source: >> x300_core/bus_int_i/rfnoc_sandbox_i/b_X0_6/noc_shell_X_i/pulse_stretch_min_ce/state_reg_replica/C* >> * (rising edge-triggered cell FDRE clocked by >> ce_clk {rise@0.000ns fall@2.333ns period=4.667ns})* >> * Destination: >> x300_core/bus_int_i/rfnoc_sandbox_i/b_X0_6/noc_shell_X_i/ctrlport_endpoint_i/gen_async_fifos.out_fifo_i/o_rst_sync_i/synchronizer_false_path/stages[0].value_reg[0][0]/D* >> * (rising edge-triggered cell FDRE clocked by >> bus_clk_div2 {rise@0.000ns fall@5.333ns period=10.667ns})* >> * Path Group: bus_clk_div2* >> >> The timing constraint slack is happening in my X block noc shell, most >> certainly because of what I am doing in my verilog design. However, because >> the bit file "works" as far as the output I am expecting, I made a design >> decision to continue on while not making the constraint. All of my useful >> work is done in IQ, and the noc shell slack seems to be on the ctrlport. >> >> I modified the script to continue on success and record success/failures >> to a csv. I am getting 19 failures and 11 successful builds over the 30 >> runs I did. >> >> [image: image.png] >> >> My next set of 30 runs will not change the build seed to get more data. >> >> Thanks, >> >> David >> >> >> On Thu, Mar 13, 2025 at 10:51 AM Sam Lane <sam.lane@surrey.ac.uk> wrote: >> >>> Hi David & Wade, >>> >>> I would just like to chime in on this that I've been having exactly the >>> same issue, repeatably, building for N310&320 series devices. I'm running a >>> heavily customised image with plenty of (known good) custom logic, and the >>> error comes up seemingly at random following a commit. >>> >>> The only (semi-) repeatable way I've found of fixing this is renaming >>> the image_core.yml file from which rfnoc_image_builder is running. Strange, >>> I know, though it seems to work 50% of the time, at least for me. If you >>> find any tricks other than those previously mentioned please let me know. >>> >>> I'm going to do some digging once I'm not under deadline-pressure, as >>> this issue is getting on my nerves somewhat. >>> >>> Kind Regards, >>> Sam >>> >>> >>> ------------------------------ >>> *From:* Wade Fife <wade.fife@ettus.com> >>> *Sent:* 06 March 2025 17:01 >>> *To:* David <vitishlsfan21@gmail.com> >>> *Cc:* USRP-users@lists.ettus.com <usrp-users@lists.ettus.com> >>> *Subject:* [USRP-users] Re: rfnoc_image_builder error: trying to tool >>> lock on already tool locked arc >>> >>> Hi David, >>> >>> I'm surprised that you're seeing it that frequently. The Ettus >>> continuous integration tests build FPGAs regularly and from what I >>> understand this issue is pretty rare there. This makes me wonder if there's >>> something about the images you're building that causes this to reproduce >>> more frequently for you. Can you estimate what percentage of unique builds >>> (with a unique build seed or git hash) fail? Which FPGA image are you >>> building? Does it have custom logic in it? >>> >>> You could use the repeat_fpga_build.py script to automate building the >>> FPGA multiple times to get a successful build. It automates the process of >>> selecting a unique seed for each build and can even run multiple build jobs >>> at a time. >>> >>> Thanks, >>> >>> Wade >>> >>> On Mon, Mar 3, 2025 at 1:30 PM David <vitishlsfan21@gmail.com> wrote: >>> >>> Using UHD 4.6/Ubuntu 22.04/x310, I have built many images in the last >>> year or so with rfnoc_image_builder. Recently in the last month, I get the >>> following error on almost all images: >>> >>> [image: image.png] >>> >>> I have tried the following, after referencing this the known issues >>> section in the USRP3 build instructions ( >>> https://files.ettus.com/manual/md_usrp3_build_instructions.html): >>> >>> 1. doing the suggested and making a non-functional source code >>> change and recommitting the git >>> 2. deleting the .git directory in both the block directory and the >>> uhd/ directory where the fpga build happens >>> 3. changing the build seed in uhd/fpga/usrp3/top/x300/Makefile >>> 4. Running on a different machine, copying the block source code and >>> using a different UHD git all together (private rehost vs the github UHD). >>> The vivado 2021.1 install is the same as its on a network file system >>> >>> These do not produce repeatable good results. Maybe once a week or once >>> every two weeks one of these things will finish the build. This has been >>> happening for about a month or two, and I don't know how else to >>> troubleshoot. >>> >>> Any advice? >>> >>> Thanks, >>> >>> David >>> _______________________________________________ >>> USRP-users mailing list -- usrp-users@lists.ettus.com >>> To unsubscribe send an email to usrp-users-leave@lists.ettus.com >>> >>>

1.  doing the suggested and making a non-functional source code
change and recommitting the git
2. deleting the .git directory in both the block directory and the
uhd/ directory where the fpga build happens
3. changing the build seed in uhd/fpga/usrp3/top/x300/Makefile
4. Running on a different machine, copying the block source code
and using a different UHD git all together (private rehost vs the github
UHD). The vivado 2021.1 install is the same as its on a network file system

Wade and Sam, The issue has been resolved by replacing my local copy of timing.xdc with the remote UHD-4.6 version. 100% success over 30 builds. I messed up the file in the past 2 months, probably by doing something with the Vivado project settings when I was doing a debug ILA image. Thanks very much for the help! David On Sat, Mar 15, 2025 at 1:27 PM David <vitishlsfan21@gmail.com> wrote: > Wade, > > Took a look at the timing.xdc you sent, and mine is distinctly different. > It also contains custom block names I've used in the past that have nothing > to do with the design. No modifications were made to this file on my end. > > Searching for the "## Asynchronous Paths" comment on line 597, my local > timing.xdc looks like: > > [image: image.png] > > It appears to be manually listing all the blocks, instead of the > convenient star notation I see in the linked timing.xdc line 599. > Interestingly, the name of the block I am working on "X" does not appear > anywhere, only old block names made a long time ago. > > I'm not sure why the old block names are in there, or why the format of > the timing.xdc file looks different. I will make a backup and use the file > on the remote UHD-4.6. > > Thanks, > > David > > > On Sat, Mar 15, 2025 at 1:09 PM Wade Fife <wade.fife@ettus.com> wrote: > >> Thanks David, that is helpful. I suspect that this tool lock bug shows up >> more frequently when Vivado is struggling to meet timing. If you fix that >> timing issue then the problem might become less frequent. It looks like >> that path is on a reset going to a "synchronizer_false_path", which should >> be a false path (hence the name). A false path means that Vivado should not >> be doing timing analysis on it and can make it as long as it wants. There >> is a constraint here that should be setting that as a false path: >> >> >> https://github.com/EttusResearch/uhd/blob/UHD-4.6/fpga/usrp3/top/x300/timing.xdc#L599 >> >> set_false_path -to [get_pins -hierarchical -filter {NAME =~ >> */synchronizer_false_path/stages[0].value_reg[0][*]/D}] >> >> Did you by chance modify the timing constraints? Or the file >> Makefile.x300.inc where that constraint gets pulled in? >> >> Wade >> >> On Sat, Mar 15, 2025 at 12:48 PM David <vitishlsfan21@gmail.com> wrote: >> >>> Wade and Sam, >>> >>> The repeat FPGA build script was very useful. I was able to let it run >>> and find a build solution. That will let me proceed with my project, but I >>> still want to nail down why this showed up. >>> >>> Some additional info: my design does not meet timing constraints, but it >>> still builds the bit file. This has been the case for some time before this >>> tool lock error came up. The bit file still works for my use case, so a >>> failure for timing is still a success to me. The error in the >>> post_route_timing_summary.rpt is: >>> >>> *Slack (VIOLATED) : -0.948ns (required time - arrival time)* >>> * Source: >>> x300_core/bus_int_i/rfnoc_sandbox_i/b_X0_6/noc_shell_X_i/pulse_stretch_min_ce/state_reg_replica/C* >>> * (rising edge-triggered cell FDRE clocked by >>> ce_clk {rise@0.000ns fall@2.333ns period=4.667ns})* >>> * Destination: >>> x300_core/bus_int_i/rfnoc_sandbox_i/b_X0_6/noc_shell_X_i/ctrlport_endpoint_i/gen_async_fifos.out_fifo_i/o_rst_sync_i/synchronizer_false_path/stages[0].value_reg[0][0]/D* >>> * (rising edge-triggered cell FDRE clocked by >>> bus_clk_div2 {rise@0.000ns fall@5.333ns period=10.667ns})* >>> * Path Group: bus_clk_div2* >>> >>> The timing constraint slack is happening in my X block noc shell, most >>> certainly because of what I am doing in my verilog design. However, because >>> the bit file "works" as far as the output I am expecting, I made a design >>> decision to continue on while not making the constraint. All of my useful >>> work is done in IQ, and the noc shell slack seems to be on the ctrlport. >>> >>> I modified the script to continue on success and record success/failures >>> to a csv. I am getting 19 failures and 11 successful builds over the 30 >>> runs I did. >>> >>> [image: image.png] >>> >>> My next set of 30 runs will not change the build seed to get more data. >>> >>> Thanks, >>> >>> David >>> >>> >>> On Thu, Mar 13, 2025 at 10:51 AM Sam Lane <sam.lane@surrey.ac.uk> wrote: >>> >>>> Hi David & Wade, >>>> >>>> I would just like to chime in on this that I've been having exactly the >>>> same issue, repeatably, building for N310&320 series devices. I'm running a >>>> heavily customised image with plenty of (known good) custom logic, and the >>>> error comes up seemingly at random following a commit. >>>> >>>> The only (semi-) repeatable way I've found of fixing this is renaming >>>> the image_core.yml file from which rfnoc_image_builder is running. Strange, >>>> I know, though it seems to work 50% of the time, at least for me. If you >>>> find any tricks other than those previously mentioned please let me know. >>>> >>>> I'm going to do some digging once I'm not under deadline-pressure, as >>>> this issue is getting on my nerves somewhat. >>>> >>>> Kind Regards, >>>> Sam >>>> >>>> >>>> ------------------------------ >>>> *From:* Wade Fife <wade.fife@ettus.com> >>>> *Sent:* 06 March 2025 17:01 >>>> *To:* David <vitishlsfan21@gmail.com> >>>> *Cc:* USRP-users@lists.ettus.com <usrp-users@lists.ettus.com> >>>> *Subject:* [USRP-users] Re: rfnoc_image_builder error: trying to tool >>>> lock on already tool locked arc >>>> >>>> Hi David, >>>> >>>> I'm surprised that you're seeing it that frequently. The Ettus >>>> continuous integration tests build FPGAs regularly and from what I >>>> understand this issue is pretty rare there. This makes me wonder if there's >>>> something about the images you're building that causes this to reproduce >>>> more frequently for you. Can you estimate what percentage of unique builds >>>> (with a unique build seed or git hash) fail? Which FPGA image are you >>>> building? Does it have custom logic in it? >>>> >>>> You could use the repeat_fpga_build.py script to automate building the >>>> FPGA multiple times to get a successful build. It automates the process of >>>> selecting a unique seed for each build and can even run multiple build jobs >>>> at a time. >>>> >>>> Thanks, >>>> >>>> Wade >>>> >>>> On Mon, Mar 3, 2025 at 1:30 PM David <vitishlsfan21@gmail.com> wrote: >>>> >>>> Using UHD 4.6/Ubuntu 22.04/x310, I have built many images in the last >>>> year or so with rfnoc_image_builder. Recently in the last month, I get the >>>> following error on almost all images: >>>> >>>> [image: image.png] >>>> >>>> I have tried the following, after referencing this the known issues >>>> section in the USRP3 build instructions ( >>>> https://files.ettus.com/manual/md_usrp3_build_instructions.html): >>>> >>>> 1. doing the suggested and making a non-functional source code >>>> change and recommitting the git >>>> 2. deleting the .git directory in both the block directory and the >>>> uhd/ directory where the fpga build happens >>>> 3. changing the build seed in uhd/fpga/usrp3/top/x300/Makefile >>>> 4. Running on a different machine, copying the block source code >>>> and using a different UHD git all together (private rehost vs the github >>>> UHD). The vivado 2021.1 install is the same as its on a network file system >>>> >>>> These do not produce repeatable good results. Maybe once a week or once >>>> every two weeks one of these things will finish the build. This has been >>>> happening for about a month or two, and I don't know how else to >>>> troubleshoot. >>>> >>>> Any advice? >>>> >>>> Thanks, >>>> >>>> David >>>> _______________________________________________ >>>> USRP-users mailing list -- usrp-users@lists.ettus.com >>>> To unsubscribe send an email to usrp-users-leave@lists.ettus.com >>>> >>>>