This will detail what will go inot the pi05 -franka page.

For the setup, here’s what I want.

This section describves the robot setup we have. A layout is hsown in Fgiure 1.

Now, Figure 1 shows a franka robot arm (make it do a nice motion), connected to a control pc.

Then there’ s layer called the Frnka control interfrace (FCI), and that communicates with a NUC PC, through polyumetis.

The worskation also has camera, and the caemra are connected to another machine called the GPU machine , which takes in the cmara obseravations and the robot states. The GPU takes to the NUC using the polymetis.

The GPU machine send the current obs, langauge command and robot states, over to a nother machine, called the server machine. the server machine, runing pi0.5, then respodns back with the action chunk.

Make a nice motion, of like how a message flows through the system.

Make Figure 1 a bit slower (the message move too fast). I dont like the shape of the camera, instead, draw a silloute of 2 D405 and 1 D435 realsewsen camear.

For the machines, make them look like computers. the control pc of franka looks like https://frankarobotics.github.io/docs/overview.html here (See control in the scehmatic).

When the server machine recives the mesages, show like a diffusion process starting from a random action trajctory and like converging to a nioce smooth one, similiar to how the diffusion head of pi0.5 genertta action.

Instead of sectuoi 2. Finetuning, i want to call it Dat collcetion.

Here’s a rough idea of text. To collect demonstration data, we use a meta quest 2 VR headset. we use the repo for droid,

and the respoective https://github.com/droid-dataset/droid/blob/main/droid/controllers/oculus_controller.py.

Then Figure 2 is a scheamtic.

In figure 2, I want to show a sillouete of a human holding the quest vr contreooler, andthe VR headset sitting infront. Then, i want to show that the delta motion of the contreooler is coped at the end effrector of the franka robot.

Now, i want to add something about freqquerncy. A new paragraph. “one thing thast’s important in democollection is what frequency to collect at”. Realistically, your frequency will depend on many factrors, e..g your harware setup, your tasks, your server, your network delays etc. The first thing that direclty hits you for freqwuency is size.

In a standradrd VLA demo collection setup, we have 3 realsense cameraa. “AGENT: find the default fps for ther cameras we have the d405 and d435 and their default resolution”. In figure 3 below, show sliders where the fps is at default (maybe 60 fps), and the data freqwuerncy slider, which then shows an estiamte of the dataset size (also slide for length of the task), and slider for total number of demosntrrations.

In section 3 the deployment on the franka,

i want to say that one important thing is matching depolyment speed with data speed. Say that in most of our data, we collect at somewhere between 8 and 15 hz, because we dont have much storage.

To get deployment speedd, you must find how fast you can send a set command to the robot. Here’ should mentiuon that the main reason there’s a NUC is that franka demmands maintiantg a 1khz control frequency, so typically a NUC is placed running ubunutu in rtc (figure out what the real time compute setting sdoes, maybe add a footnote).

In our setup, a sending a env.setep(Action) acomdn freom the GPU machine takes about 3.4 ms on average, (so the frequency is …). Say how much a sigle demo for 1 min at that frequnecy would be size wise.

Now,anything beyond that, the way to set the frequency tro a desrired. frequency (say 8 hz to match data) is to spend 3.4 ms, the communcation, and then force a sleep equal to the desried period minus the 3.4 ms elapsed time.

Now, the reason this is important is the following.

I want a fgirue showing the robot arm passing through waypoints in space, and showing 1/8 sec (125 ms) time between them. That will be the left of the figure. saying demo collectiuon. on the right oif the figure, dasying deplouyment, i want to show various setting, based on a knob the user can play with. If the depoilyment is 8 hz, showing matching motion. If the deployment hz is faster or slower, show how that will affect the robot motion.

If the robot learned that going from point A to point B should take 125 ms, we receive an action chunk doing that. But when we run that chunk at a higher frequency, the robot will achieve the same motion in faster time, showing how slowly the robot drifts. same for slower.