Forums Niryo One Troubleshooting Burst CAN connection problem with motor

Viewing 6 posts - 1 through 6 (of 6 total)
  • Author
    Posts
  • FindingNiryo
    Participant
    Post count: 63

    Every few hours V2 rpi_setup.launch logs [ERROR] CAN connection problem with motor 1,2, or 3 (plus 4 on V1).

    A string of these errors occur within a few seconds for a single motor triggering CanComm : Stop hardware control loop followed by DXL : Start hardware control loop, Set calibration flag, CAN : start hardware control loop.

    This occurs while robot is powered on but otherwise idle and requires arm to be recalibrated.

    Edouard Renard
    Keymaster
    Post count: 161

    Yes, this can happen sometimes if the CAN bus starts to be too busy, but we have managed to run several robots (idle or not, it does not matter for this test) without this kind of issue for several hours.

    This scenario should be really rare though. Chances are increasing if the cable connection is not perfect. Maybe check if the connectors are correctly fixed, I guess if you reinforce some parts of the connector that will reduce the frequency at which you get this error (and maybe you’ll never get the error again).

    FindingNiryo
    Participant
    Post count: 63

    Isn’t all traffic on the CAN bus operating at fixed update frequencies?

    I would not expect Idle robots without torque engaged to have much variation in CAN bus traffic.

    Since each NiryoStepper motor has failed on both V1 & V2 robots it does not seem likely to be a connector issue.

    Is NiryoStepper implementing random back-off on bus collision detect?

    Edouard Renard
    Keymaster
    Post count: 161

    Here are more info to let you understand how things work on Niryo One:

    – The problem with the CAN bus is that it’s a multi-master bus. So, the RPi will send commands to each motor, but also each motor will send some data (at a pre-defined frequency, and each node of the CAN bus has a unique ID). Thus, we can’t really control when exactly data is sent/received, and it’s possible that suddenly the CAN bus is saturated and thus will preempt the frames from lower priority nodes (the ones with bigger IDs).

    – Second problem is that the MCP2515 we’re using has only 2 buffers for incoming messages. So the RPi will receive data from 3 motors (4 for V1), and the MCP2515 will have to handle all that with 2 buffers, which is quite limited compared to other components that can allow more than 6 buffers.

    Conclusion: CAN is a great communication bus, but maybe not the best type of bus when working on a master/slave system (RPi master + motor slaves). Why we chose to use it ? Well because it was a much better solution than what we tried before (I2C or SPI daisy-chain), and still affordable to setup + not too complicated with the components that are available for us. (same reasoning for MCP2515)

    Hope this gives you a better understanding. And as you can see, using or not using torque is not a variable is this problem, it’s mostly about the bus itself. That’s why I emphasized the physical connector quality, which is the only thing that you can play with to get better results. (and from the tests that we do here, we almost never have a problem with the CAN bus for new robots).

    FindingNiryo
    Participant
    Post count: 63

    I continue to see idle robot arms spontaneously restart the hardware control loop (requiring recalibration).
    I understand the factors you described in the working of the CAN system, but this failure mode is quite disruptive and there are many much more demanding applications which make use of the CAN bus without such faults.
    Could you please look into the bus traffic and message handling queues of Niryo_One & Niryo_Stepper to confirm interrupts and bus collision protocols are properly implemented?

    Edouard Renard
    Keymaster
    Post count: 161

    Finally I guess that it’s more of a cable issue than a saturated CAN bus. We have continuously improved the cables for Niryo One, and the tested robots we ship now have zero connection problems when being idle. (even if we still see some CAN frames dropped, but not enough to trigger the connection problem)

    If this problem persists, please contact our technical support and we’ll see what we can do for you!

Viewing 6 posts - 1 through 6 (of 6 total)

You must be logged in to reply to this topic.