Thank you for your quick reply, Prof Taylor! At first, the diagonal terms of F are seems correct in the beginning. Then they got unreasonable towards the point when the analysis aborted. This might be caused by the non-diagonal terms.
1. At each step, it does converge to zero, but the convergence rate is linear. For example, after 7 Newton loops, the residual norm reaches to 1E-7, energy norm got to 1E-14.
2. The problem should be well-posed, and the element did work well with other material models. I am not sure about the details of the element. Can we check the order of the element?
3. For this material, we don't have any user-defined history variables. I used the "show dict" command after running a one element test with a set of material parameters which works (it does work with some parameters), and I found that the length of the "H" array is 4, and parcn of "H" array is 2, array number is 49. I am not sure how to check the system history variables. I tried "show H", it outputs "Terms in array H are zero". Does this mean no history variables were passed between the load steps?