This prop piece was used in Season 4, Episode 19 (titled Sweet Dreams). This was episode had the final appearance of Finn (not in a flashback) and performance by Finn. It is seen in various locations during a montage scene where Finn is discussing college.

These are seen on extras as Finn is being surrounded by large groups of people dancing around and ‘partying’ while he is attending the University of Lima. This school was actually filmed at Occidental College in Los Angeles, CA. You can see several extras sporting these foam fingers in the fountain of the university.

It is unclear if these were used during filming but highly believed. They did come with a Certificate of Authenticity so they are from the props department of the show and the Warner Brothers Studio.