It's not that Wudang's Silkworm Skill is afraid of your Fatal Skill, but Yun Fei Yang has left Wudang.
6 seconds sound clip from the Return of the Bastard Swordsman movie soundboard.
You can hear this line at 00:08:44.199 in the DVD version of the movie.
Quote context
[...]
- You've come to pay us a visit. I apologize for not welcoming you. Please forgive us!
- May I ask the purpose of your visit? What can we do for you? Yes?
- Tell Yun Fei Yang to come out.
- Our Master has attained the tenth level of Fatal Skill. We're here for a duel!
- Go get him! Is he too scared to come out? What's wrong? Is he too scared to come out?
- Please come again in seven days.
- Very well! I will be back...
- If Yun Fei Yang is not here by then, I'll slaughter everyone in Wudang!
- But... During the next seven days no one is allowed to leave, or else they'll be killed on sight!
[...]