Workflow/pipeline ideas

Starting a thread on how we each work in pyr (altho I imagine same workflow method(s) may be followed in BANC/FW and/or new-er datasets not yet out), hoping that maybe we can each inspire each other to adopt/improve/adapt on idea(s) from how we approach working in these games/datasets.

This is my approach to dealing with merger fest origin segment(s), taking on/adapting from creating farms in FW/BANC (altho for different reasons), so in 1st pic:


its the origin segment after i’ve completely un-mergfied it.
in 2nd pic:

are the proofread segments, hidden ones are the finished ones, visible one currently being worked on. (copy pasting coordinates)

like that I dont have to worry abt which of the hidden ones i’ve completed and which ones need to be worked on yet. When there’ll be no more stuff to be proofread from origin segment I’ll paste the 2nd pic’s link in the gsheet with everything visible.

Can’t wait to see all y’all’s approaches to proofreading/tracing stuff. :grin:

4 Likes

If i have many segments, i mark all completed cells by changing the colour of them so they get a border around the colour and hide them. If the rest of the cells make a mess i hide all cells except the one i am working on. To limit the number of segment i try to split one segment or cluster of segments at a time from staring piece
I have the starting segment on the upper board and turn this on and of at the end to make sure i have traced all part of it.
I also merge other cells if i find their connection during the work, but are not following them more than i have to make sure they are not part of the starting segment. And try to clean up the workspace for each finished segment if i can.
i also have everything visible when pasting link, unless i need a second option where only the segment in question is visible.

3 Likes

I’m starting by splitting all the initial mergers. Then the first step in my workflow is to mark all the ends (in all the segments), that reach the end of the dataset. This way I have a lot less work (at least, visually, lol). After that, I’m working first with “vertical” and smaller segments to eliminate them from the big picture quicker.

2 Likes

interesting but dont all of the neurons/branches confuse you? Or do the annotations help you know what/where you’ve proofread?

1 Like

It might be confusing sometimes, but then I merge all the marked segments and, if need be, refresh the site, look quickly through the scene and find all the branches still not annotated. Rinse and repeat.

2 Likes

Great idea for a thread, always nice to see others’ workflows and get more ideas! Helpful for us to see how people interact with the interface as well.

3 Likes

Love this idea, I’ll actually try to record or fully describe my process this weekend.

3 Likes

For my current hell-segment, which I think will finish with about 3-4 dozen cells, I have the main chunk and whatever I’m currently working on visible and everything else hidden. I make it a point to restore all of my cells when I have a branch completed and then remove any remaining starred segments. I will cut segments as soon as I see the need, but I will usually wait until I’m done with a branch, temporarily or completely, before I start merging segments together.

Often times I’m hiding things that aren’t done, my plan is to check them one by one if I ever start running out of obvious things to work on.

Most segments though I’m simply starting at the biggest branch point in my segment, making sure it’s not a merger, and then just working toward whatever side of the dataset is closest to mark a branch done. I don’t need to hide much.

3 Likes

I will say also, I’m prone to frequently switching the aforementioned hell-segment from Needs Help to WIP and back depending on if I’m actively working on it. There’s… so much still to do and I like breaking it up with sane normal tasks for a quick break!

3 Likes

Here’s the methods I’ve been using.

Starting out

► I always work from one of my own old neuroglancer links, because they have settings tweaked the way I personally like them (here’s a fresh start for example: neuroglancer) Honestly I can’t recall everything I’ve tweaked here, other than turning off skeletons and one of the two border lines.

► After refreshing the starting segment (unselecting & reselecting in 2D), I immediately go after breaking apart all of the mergers so that every segment is separate before I start any other edits.

  • I start by going after the most obvious mergers, and then checking every branch off.
  • If the initial segment was a crowded mess, I will focus down small sets (get one chunk split off, then split that chunk down further).
  • Once I have done a simple visual check of a split segment, I will hide it with the eye icon as I get the rest pruned apart.

Proofing All Segments

► I start by checking the far ends of each segment, extending until I hit a data set edge or end of branch, and then do a quick annotation mark (usually one or two slides further ahead so it’s more visible in 3D). Now using KK’s wonderful Ender addon for this so that I don’t have to jump tabs in neuroglancer.

► If you’re new to neuroglancer - you’ll learn that you can trust the system to trace the basic neuron flow really well. It’s a lot less about merging every bit of dust and checking for where segments should, or shouldn’t, be. Because of this, as I am extending (pre-merge), I right click jump to the end of each segment, rather than scrolling through.

► As I am extending, I wait to merge until I have hit the end of that extension. Then, after marking the end of the branch/data, I merge backward to my starting point. As I do this, I click through the 3D (right click) to jump along the segments. This gives me a second chance to check any funky shapes along the branch.

► After checking all ends, and taking a quick zoom out to make sure no branches were forgotten, I will dim the color of the branch (changing the color is a great idea, just took this from annkri’s earlier post!) and hide it with the eye icon, then move on to the next segment. It makes it much easier to tell which segments I have completed, as dimming the color not only highlights the color box but also selects a color that the software doesn’t choose. It also helps in a later step. Ex:

image

► I use a single annotation layer. I mark ends (single dot), visual path trace through a tough spot (line annotation), and for areas of interest or concern, I will use the box tool.

► When I’m done with all segments, I turn on all the segments I’ve traced. Then I will toggle on the original segment from the top box and compare it to the segments I have. If when flipping it on and off I find a segment got lost to the void while I was working, I’ll use this to find it and add it back to the pile. Ex of this: neuroglancer

Alternative to handling lots of segments

While I haven’t had as many mega mergers as some of you have had the bad luck for, an alternative I keep in my pocket for when I do is to, instead of ‘hiding’ the finished segments, take their ID number and add them to the upper box list (this is how I keep myself sane with major farms in the BANC).

Here’s how that might look: neuroglancer

Having the list of IDs tucked in that part of segmentation means that you will not lose them unless you delete that list of IDs in the textbox, which makes it ideal for managing a lot of segments you don’t need to keep open.

3 Likes

I’ve finally found a larger merger, and @annkri’s method of changing the colors of already completed ones was really useful. Thanks! I’ve selected black since it’s easy to consistently choose the same color in the color selector. I’m hiding them anyway, so it doesn’t matter much. Just a small tip: if you’re done with all the segments, you can right-click on the color selector to remove the selected color and reset it. This will allow Neuroglancer to randomly select a color and change it when the seed is modified (when you press the ‘L’ key on your keyboard).

2 Likes

A mess some starter merger branches by hidding them with the star. How can i undo to get the branches back ?

3 Likes

one way is to get the initial segment from the gsheet and add it back in, it still exists as spelunker/cave isnt like EW, nothing is lost including all history/past/pre-split segments. And from there extrapolate which segments youre missing.

Another way is to email/dm someone like M and ask her if she can do some back-end magic, pull the segments youve edited the last 24-48 hrs and give you the IDs. If you’ve edited them prior to mistakenly star clicking them you should have them back.

3 Likes

“one way is to get the initial segment from the gsheet and add it back in” fabulous tip i did not think about ! Thank you :vulcan_salute:
Edit : I will check all my previous cells to validate if i missed some. I’m not used to reap multiple cells in the same one, like Eyewire/Mystic so i certainly “unstar” some by habit.

3 Likes

yeah it takes some getting used to. Good luck.

2 Likes

I have found that if you have a nice contrast colour of the starter segment this will make it easier to see what is missing. the way i am working i am usually selecting a colour close to the auto selected colour for any complete branch, and then find that having the starting segment white is really useful. I am guessing this would be even easier to see if the cells are a bit on the dark side.

3 Likes

It’s work like a charm ! I’m able to recover some “unstared” starter merger ! :smiling_face_with_tear:

2 Likes

Same to me, flashy colored help a lot. I work a lot with hide function also to keep untouched branch out of context. And i always let every single seg until the end, for a better view of WIP segs.

2 Likes

As i don’t really know where to put this information, i write it here.
I just realized a very useful trick when I can’t cut a segment with a “failed 400” error. I took a random segment and merged it to the one that is “failed 400”, and when the line is refreshing, I had the pleasant surprise of seeing that refreshing the segment automatically cut what I was trying to do previously. I don’t know if you understand the logic. I don’t have the example on hand to make you screenshots but I’ll think about it next time to show you and you can tell me what you think.
Basically, a merger that cannot be cut → pasting of any piece → the segment is updated without the previous merger.
This has already happened to me twice. Of course the new one without the merger is a copy. Have you ever had this phenomenon?

2 Likes

if i understand what you mean i have had segments where i try to cut of mergers, where the segment is not updating, but if i click on it, in 2d to remove it from sight, clicking on it again will give a new updated segment.

1 Like