Skip to content

Bianca Introductory Workshop QaA

tags: UPPMAX workshop

Link to this document: https://hackmd.io/@UPPMAX/Bianca_Intro_QaA

Welcome!

Course material: https://uppmax.github.io/bianca_workshops/intro/

Course project number sens2023598

:::warning ⚠ Don't share any sensitive information in this document! :::

Log in to Bianca

  • Q: WHat is 1 + 1?
  • A: 2
  • A: Start with one finger up then for each +1 raise one finger, how many fingers are you holding up when you are done? (example of the type of answer you migth get)

  • Q: How many users and who do you share a node with?

  • A: You share "virtual login node" with everyone in the project. That is for Bianca just 2 cores!!
  • In this course it is thus quite crowded
  • So using compute nodes is necessary for real jobs
  • A: If you are alone in a project you can get far with doing much on the "login node"

  • Q: Can I use my local thinlinc or is it only the web-version on Bianca?

  • A: Just web version! We were a little unclear.
    • A: [Richel] it should be clear in the documentation, if not, please help us improve! - no it was quite clear there but from the talk earlier I thought maybe there was a way to do it (and I've used the app version for Rackham and it works well I think, better than command line). [Richel] Great! I am happy it works well got you :-)
  • A: For our other cluster (Rackham) you can do with the ThinLinc App.

  • Q:So if I'm working on my own doing mostly small stuff and I'm satisfied with the performance of R, I can use the "login node"?

  • A: It is not good practice to use the "login nodes" for calculations or analysis,e.g. in R performance is usually much better in a interactive session.
  • A:

  • Q: I dont understand how to log into SUNET, I am working from Lund University. I am working from home, so currently not on eduroam. I already have a VPN like this but it does not seem to work. Do I need to go to the university instead? My VPN is not with fortiClient, does it have to be?

  • A: [Richel] Are you within the university network? If yes, you are already in :-) . Your are within SUNET at Lund university if you work from the unversity 👍 (i.e. eduroam). No VPN needed :-)

  • A: [Marton] I had the same issue, this is a pretty easy-to-follow guide to solve it :-) If you already have fortiClient on your computer, you still have to add a new VPN connection with the following details: provider fortiClient, name "LU SSL VPN" (although I think it can be anything), address vpn.lu.se -> then join it with the account you join Eduroam as well LU email + password.

  • A: It can be other providers too I guess, but there are more specs to define: - Server name or address: enter vpn.lu.se - VPN type: select L2TP / IPsec with pre-shared key - Pre-shared key: enter luvpn123 - Type of sign-in info: select Username and password - Username: enter your LucatID / StudentID - Password: enter your Lucat password / Student password Anyway way fortiClient seems the easiest and it is super easy to add to your PC takes like an extra 400kb space and 1 minute. link
  • A: [Richel] Thaks Marton! I will link to it from the UPPMAX doc now, until the Lund URL breaks
  • A: Check here https://www.maxmind.com/en/locate-my-ip-address and make sure you see "Swedish University Network,..." in your "ISP / Organization".
  • Q: Is there a project on bianca for this workshop we should log in to or should we use the project login to our own respective projects? Thank you!

  • A: sens2023598

  • A: [Richel] my bad! Pick whatever project you like, yet sens2023598 are resources reserrved for the project (so you do not eat up your own computing time)

Feedback on this lecture and exercises

For those that are ready. You are encouarged to say both things that are good and things that can be improved :-)

(Feedback can be anonymous, you pick!)

  • Feedback 1: In the email with the pre-course instructions you didn't mention that one needed to have VPN connection. Of course, on the Uppmax pages I could figure it out, but maybe it could be added in that one paper document with things to do before the course
  • Feedback 2:
  • Etc.

The command-line on Bianca

  • Q: How is computing time counted? Am I using computing time while I (and not the computer) am thinking?
  • A: It accounts only the time you were running on allocated compute node/CPU Not the time you asked for, but the time the job took running.
  • But if you are finished with you work in an interactive session, if not ending that session the core hours will be ticking!
    • More in afternoon!!
  • Q:I cannot run my script for some reason /Sigrid

  • A: did you chmod +x? yes then make sure you start with ./ before the script name. Ah, now it works! I thought if I was in the same folder as the script file it would execute anyways 👍 -

  • Q: So does scp and rsync actually work on Bianca? I thought from the materials that we can only transfer files on the sftp to wharf

  • A: More at 13:00!
  • But we have the wharf

  • Q:I can't do ctrl + v nor paste in the thinlink webversion - is there a secret trick to it?

  • A: There is a "trick". Look for tools on the left of the Thinlinc session - there is clipboard here. Annoying, but better than nothing. Ah, yes thank you!
  • A: https://uppmax.github.io/bianca_workshops/slurm_intro/#interactive-jobs and scroll down a little.
    • This should be earlier as well, but is unfortunately not
  • Q:I have no permission to create a file

    • you must be in a place where you have these rights. Like in project folder or in ypu home folder (/home/<username>)
  • Q: If I want to remove the symbolic links i made, can I go to the folder and type rm nobackup? I'm afraid to remove the actual backup instead of the link you see..

    • make sure you point to the link and not the origin and it should be OK

    • how do I make sure of that?

    • do: ls -l and check to the left if the line starts with l
    • yep it does
    • it shall also contain the redirect sign: ->

Feedback on this lecture and exercises

For those that are ready. You are encouarged to say both things that are good and things that can be improved :-)

(Feedback can be anonymous, you pick!)

  • Feedback 1: .... Great dynamics of the course! I wish I had this kind of introduction at the very beginning of my "bioinfo journey" Will certainly share to my colleagues that are starting! Thanks
  • Feedback 2: Excellent course and information, thanks!

Module system

  • Q: We can't do the bigger exercises on the login node right?

    • A: [Richel] Well spotted! You are right! I should have made that clearer, thanks for helpong me see that! I will fix the doc now... It is fixed now :-)
    • A: gatk HaplotypeCaller ... is quite heavy so skip that step and the following ones in that exercise
      • we could make that exercise in the 14:00 session if there is time
      • It is already there!!

    • A: RStudio WILL be tested in the 14.00 session
    • A: Conda exercise is fine but that regards only loading the module!
  • Q: How can one request to install a module with newer version?

    • A: Use our support system. Go through supr.naiss.se
    • An application expert will probably install it, but depending on the software it may take longer time.
    • A: You can also to some extent do own installations. We cover that in the intermediate Bianca workshop May 24.

Feedback on this lecture and exercises

For those that are ready. You are encouarged to say both things that are good and things that can be improved :-)

(Feedback can be anonymous, you pick!)

  • Feedback 1: Great lecture and exercises that helped me a lot! I also liked the style of teaching since later we have to figure out the steps for ourself
  • Feedback 2: ....
  • ...

Transferring files

  • Q: Maybe this is out of scope from the course, but what will be the main/practical differences of transferring files with scp, sftp or rsync? Thanks!

    • A: [Richel] Agreed, it is out of scope, thanks for the thoughtfulness! Here is a summary: scp will never work on Bianca (due to 2FA). sftp is great, terminal only, maybe a bit clumsy for folders. rsync is great, yet terminal-only (hence less friendly to beginners) and a bit more complex then sftp
    • A [Richel]: the documention at http://docs.uppmax.uu.se/cluster_guides/transfer_bianca/#file-transfer-tofrom-bianca compares these in more detail :-) Enjoy!
  • Q:[Louise] Still can't get in

    • A: [Richel] what about we move to room 4?
    • A: [Richel] the password turned out to be wrong or incorrectly copy-pasted. Well, it's fixed now :-)

Feedback on this lecture and exercises

For those that are ready. You are encouarged to say both things that are good and things that can be improved :-)

(Feedback can be anonymous, you pick!)

  • Feedback 1: ....
  • Feedback 2: ....
  • ...

Job scheduler/Slurm/compute nodes/interactive

  • Q:Sorry if its not completely related...! But I liked the way to logg in to RStudio by thinlinc (interactive mode). Will it be similar procedure to log in as this, at Dardel? Thanks!

    • A: [Richel] It's OK :-) . Unsure if Dardel does the same thing, let's see if others know this...
    • A: [Björn] still we know that users really like this way to work!

    • A: [name=Pavlin] https://www.pdc.kth.se/support/documents/run_jobs/run_interactively.html (now broken)

  • Q:

    • A:
  • Q:

    • A:

Feedback on this lecture and exercises

For those that are ready. You are encouraged to say both things that are good and things that can be improved :-)

(Feedback can be anonymous, you pick!)

  • Feedback 1: ....
  • Feedback 2: ....
  • ...
  • Q:
    • A:
  • Q:
    • A:
  • Q:
    • A:

Feedback on this lecture and exercises

For those that are ready. You are encouarged to say both things that are good and things that can be improved :-)

(Feedback can be anonymous, you pick!)

  • Feedback 1: ....
  • Feedback 2: ....
  • ...

Evaluation

<------------------------------------------------------ WE ARE HERE!

This is an anonymous evaluation and will ideally be published in raw form at https://github.com/UPPMAX/bianca_workshops/tree/main/evaluations/20240419 . To do so, please do not share sensitive data here!

Below are tallies, i.e. add one character behind each to answer a question. This helps us assess how well we as teachers did expressed in a quantity. After that, there is the option for more qualitative feedback.

Thanks already for filling this in!

I am confident that I can now login: - yes: 1111qii* - no:

I am confident that I can now create executable bash scripts: - yes:21q11i* - no:i

I am confident that I can now use modules: - yes:1q11ii* - no:

I am confident that I can now do file tranfer using a GUI: - yes:q111i - no:1i*

I am confident that I can now do file tranfer using transit: - yes:11i* - no:q11i

I am confident that I can now do file tranfer using rsync: - yes:11j - no:q11i*

I am confident that I can now start an interactive session: - yes:q11ji* - no:11

I am confident that I can now submit jobs to the scheduler: - yes:q1i* - no:111i

I am confident that I now know the legal aspects of sensitive data: - yes: q - no:i

Below are the question. If needed, specify your ideas per session.

What should we keep doing?

  • keep ther overall structure and input
  • ...
  • Good interactive sessions, especially in the morning. Maybe more interactive in the afternoon
  • ...
  • Very interactive and enjoyable course structure, logical flow of topics.
  • Great with several teachers/helpers in the sessions to give individual help when needed
  • Good with exercises so we get to try what we learn
  • ...
  • Good interactive sessions
  • ...
  • teacher/helper to student ration was really good
  • ...
  • ...
  • Amazing dynamics to keep everyone on the same page. Other bioinfo workshops I have attended tend to be less "interactive" hence more difficult. Thanks!

Which section(s) scheduled enough time for exercises?

  • The morning had too much time, while the others had too little time
  • ...
  • A bit too much time in the morning perhaps, maybe some more time could be put into the afternoon to be able to actually try the file transfer for example.
  • could be less time for basic scripting and bash exercises and more time for file transfer and job sceduling. We had the sample data, but never did not have time to properly go through it, besides reviewing the example script.
  • The morning session had enough time (perhaps optional extra exercises for the fast students)
  • Logging in from Rackham to Bianca is unclear a bit!
  • ...
  • It would have been good to spend more time on the second part, maybe to actually try a data transfer + use that data in a job scheduler
  • ...
  • ...
  • ...
  • ...
  • ...
  • ...

What should we improve?

  • more time for file transfer and exercises in that section
  • agree with the comment above
  • Maybe have some session about how to apply for an account, it is really hard to do if you've never worked on any cluster before ie estimations on compute space needed and nodes etc.
  • More time for the sessions with Björn and Pavlin in the afternoon
  • The afternoon sessions were too quick for a beginner in coding. Jargon, terminology, quickly using commands that I was unfamiliar with made me get lost pretty quickly.
  • Perhaps more info on developing a script, what way can you scale up your project and estimate workload or test running time.
  • ...
  • Don't have a feedback/evaluation document where text can (accidentaly) be deleted and edited by others :)
  • Perhaps to extend more time for the practical exercises. And go throught them together in some of the cases.
  • ...
  • ...
  • ...
  • ...
  • ...

Other comments?

  • file transfer from bianca would be nice to learn
  • More time on transfer of files, to and from desktop and also to and from other secure environments such as Vesta
  • Really great intro, it felt easy to ask questions even though it was on zoom.
  • Overall very nice course, thank you!
  • Excellent course, thank you greatly to the organizers and teachers!
  • ...
  • ...
  • ...
  • ...
  • ...
  • ...
  • ...
  • ...
  • ...

:::warning ⚠ Don't share any sensitive information in this document! :::