While my teams are still in the process of trying out, it is easiest to identify the extreme cases on either end: those at the top end of the talent spectrum, and those at the bottom. In my experience, once you've identified those players, you should place them aside. Literally, you can do this by placing them in their own groups so you can focus on diving deeper into perhaps the middle 60% or so. If you wish to take a less direct method, you can make notes on which players require little to no more observation, and instead focus your energy on players on the fringes of making the team.
I usually begin tryouts in a very hands-off manner, meaning little to no coaching beyond explaining a drill. I prefer to see what a player's natural tendencies are. How risky their decisions are, their reactions at defensive transitions, etc. In this stage especially, the edge cases will stand out pretty quickly. To help separate the remaining players however, I've found it useful to see how they react to being coached.
To give a real world example, I ran a 3four3 style 4v0 and 4v1 rondo at my most recent tryout. The main things I wanted from the players were receiving passes across their body, hard passes played to the correct foot, and proper movement to offer a passing angle. Very few players understood instantly after the example, and most needed extra coaching.
In the 4v0, it's black and white which foot the players should be using to receive, and also fairly obvious to the players what a good pass looks like. The players with quality technical ability managed this with few problems. The way I chose to 'test' was calling them out as either individuals or groups to see how they respond. Some players received praise, some received criticism, some received encouragement, and some received a little extra individual coaching. It's important to see how different players respond to each of these. If a player doesn't respond positively to criticism or individual coaching, then they will not be a very good fit in my coaching environment. Additionally, they players who responded to praise by relaxing and dropping their level will not fit into the culture I want to establish either.
I will continue this post tomorrow in part 2.