Answering to myself:
bbp_buddypress_loaded fires before everything is, in fact, fully loaded. bbPress’s component-specific extensions aren’t loaded until bp_init:7 (ty @boonebgorges for this explanation).
Changing add_action to bp_init with a priority of 8 did the job !
This may work for Site Activities, Members and Group activities
function bpfr_remove_activity_dropdown_label() {
// Remove forum filters in site wide activity streams
remove_action( 'bp_activity_filter_options', array( bbpress()->extend->buddypress->activity , 'activity_filter_options' ), 10 );
// Remove forum filters in single member activity streams
remove_action( 'bp_member_activity_filter_options', array( bbpress()->extend->buddypress->activity , 'activity_filter_options' ), 10 );
// Remove forum filters in single group activity streams
remove_action( 'bp_group_activity_filter_options', array( bbpress()->extend->buddypress->activity , 'activity_filter_options' ), 10 );
}
add_action( 'bp_init', 'bpfr_remove_activity_dropdown_label', 8 );
