Jump to content

Talk:Myers–Briggs Type Indicator

Page contents not supported in other languages.
From Wikipedia, the free encyclopedia
Former featured article candidateMyers–Briggs Type Indicator is a former featured article candidate. Please view the links under Article milestones below to see why the nomination was archived. For older candidates, please check the archive.
Article milestones
DateProcessResult
February 3, 2005Featured article candidateNot promoted
March 25, 2006Featured article candidateNot promoted
Current status: Former featured article candidate

This article was written with extreme bias

[edit]

The contributors to this article seem to be seeking out faults with the MBTI personality typology, rather than defining what it is. Articles written with such extreme bias are not common on Wikipedia and I found it extremely off-putting. I have found the MBTI typology extremely useful and tremendously accurate in interpersonal relationships. 97.82.125.50 (talk) 02:30, 15 March 2024 (UTC)[reply]

Your personal experience with MBTI doesn't impact how reliable sources describe it. MBTI has no empirical evidence supporting it.
I'm an INTP EvergreenFir (talk) 04:43, 15 March 2024 (UTC)[reply]
So, every article on Wikipedia needs to have “empirical evidence supporting it”?
Dreams are pseudoscientific.
Love is pseudoscientific
Pizza is pseudoscientific.
This is an encyclopedia, not a scientific journal.
The article needs to describe the definition and purpose of the subject, not just it’s critical opposition. 136.29.86.181 (talk) 05:30, 11 April 2024 (UTC)[reply]
Pizzas are not pseudoscientific. But if they were, it would need to be mentioned prominently per WP:PSCI. Bon courage (talk) 06:21, 11 April 2024 (UTC)[reply]
What is the experimental hypothesis for a pizza then?
My point is that there are many social constructs that are “pseudoscientific”.
The term is mainly used by adherents of scientism who don’t even understand the purpose and function of the scientific method. 136.29.86.181 (talk) 17:25, 11 April 2024 (UTC)[reply]
Follow the sources, and don't make the WP:BIGMISTAKE. Bon courage (talk) 17:32, 11 April 2024 (UTC)[reply]
I am not a Wikipedia newbie.
Sources support an article, but don’t describe how to arrange themselves in order of relevance.
Even ChatGPT would do a much better job at writing this article:
The Myers-Briggs Type Indicator (MBTI) is a psychological instrument designed to measure and categorize individuals' personality types. Developed by Katharine Cook Briggs and her daughter Isabel Briggs Myers, it is based on Carl Jung's theory of psychological types. The MBTI aims to identify how individuals perceive the world and make decisions through a questionnaire that classifies personalities into 16 distinctive types. These types are determined based on four dichotomies: Introversion (I) versus Extraversion (E), Sensing (S) versus Intuition (N), Thinking (T) versus Feeling (F), and Judging (J) versus Perceiving (P). Each personality type is represented by a four-letter code, reflecting the individual's preferences across these dichotomies. Despite its widespread application in areas such as career counseling, team building, and personal development, the MBTI has faced criticism from the scientific community regarding its reliability, validity, and lack of empirical support. Nonetheless, it continues to be a popular tool for personal and professional assessment. 136.29.86.181 (talk) 17:49, 11 April 2024 (UTC)[reply]
I am not a Wikipedia newbie. Then you should know that whataboutism is not valid reasoning here. And that AI is neither a reliable source nor can it be trusted not to invent sources. --Hob Gadling (talk) 18:00, 11 April 2024 (UTC)[reply]
And WP:CATW#7 strikes again. Bon courage (talk) 18:01, 11 April 2024 (UTC)[reply]
Thank you for pointing out the language concerns related to WP:PROFRINGE. It’s crucial to use precise and neutral terminology, especially in topics that can easily veer into contentious or fringe territory. Adhering to this guidance helps maintain the integrity and reliability of Wikipedia as a resource. Let's ensure our discussions and the article's language reflect this commitment to neutrality and factual accuracy. 136.29.86.181 (talk) 22:41, 11 April 2024 (UTC)[reply]
Understood, and I appreciate the reminder about whataboutism and the reliability of AI.
My intent is to highlight the need for a balanced representation of MBTI on Wikipedia, beyond its pseudoscientific classification. It's essential we present a full spectrum of perspectives, including its practical applications and value to many, within the framework of Wikipedia's guidelines for neutrality and sourced content. 136.29.86.181 (talk) 22:39, 11 April 2024 (UTC)[reply]
Wikipedia specifically does not do 'balanced representation', see WP:FALSEBALANCE. When the best sources are critical, so too will be the Wikipedia article. MrOllie (talk) 12:32, 12 April 2024 (UTC)[reply]
Will these sources be good enough for you to remove the controversial term "pseudoscience"? @MrOllie, @Hob Gadling, @Avatar317 etc.
[1]https://www.tandfonline.com/doi/full/10.1080/13674676.2022.2158794?scroll=top&needAccess=true
Quote:"Specific criticisms of psychological type
Incoherence
In their recent attack on Psychological Type, Stein and Swan (2019, Abstract) conclude that Type theory “lacks agreement with known facts and data, lacks testability, and possesses internal contradictions”. This harsh and sweeping criticism is perhaps justifiable if Type Theory is taken to embrace not simply the basic postulates already alluded to, but also the elaborate superstructure that has been built upon it. In Type literature, the latter is designated Type Dynamics, which is ramifying and constantly increasing in detail. It postulates complex relationships between the four dual-polarities. Added to that are elements from Jung’s writings that are equally conjectural. I submit that Type needs to distinguish more clearly between its basic postulates and the speculative material that has accreted around it in recent decades. Reynierse (2009, p. 18) agrees: It is only the eight individual MBTI preferences that have demonstrated validity – not type dynamics or the type categories formed by type dynamics. The time has come for the type community to abandon their enthusiasm for type dynamics and to discard it. A seemingly un-noticed example of internal contradictions is seen in MBTI Step II, an extension of MBTI™ in which each of the preference pairs (e.g., Extraversion/Introversion) is further divided into five subsets. Each subset is regarded as independent of the others, and each can be scored. A logical implication of this exercise is that the overall score for Extraversion/Introversion is a composite. However, this negates the core Type postulate that each individual prefers either Extraversion or Introversion (Lloyd, 2012a). These are serious and valid criticisms of Type. However, I do not believe that Stein and Swan (2019) provide convincing evidence that the fundamental postulates of Type are incoherent. They are no more so than the basic postulates of the Five-Factor model."
o0o
Even in the Dictionary of Psychology by the American Psychology Association I can't find any word stating that it's "pseudoscience", so you can't say that it has been consensually approved by most psychologist scientists that MBTI is "pseudoscience", perhaps by some colleagues.
https://dictionary.apa.org/myers-briggs-type-indicator
o0o
Even in your own source https://www.jstor.org/stable/26554264
stating that " These studies agree that the instrument has reasonable construct validity. The three studies of test-retest reliability did allow a meta-analysis to be performed, albeit with caution due to substantial heterogeneity. Results indicate that the Extravert-Introvert, Sensing-Intuition, and Judging-Perceiving Subscales have satisfactory reliabilities of .75 or higher and that the Thinking-Feeling subscale has a reliability of .61. The majority of studies were conducted on college-age students; thus, the evidence to support the tool's utility applies more to this group, and careful thought should be given when applying it to other individuals."
o0o
Another source
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10017728/#B17
Quote: "We sampled 529 participants who were graduate and undergraduate students enrolled in business administration programs from Colombian universities. Results show conclusive evidence of the psychometric measurement of both MBTI and leadership practices, even though the relationship between MBTI and the leadership practices inventory proved to be weak" NgHanoi (talk) 17:51, 21 April 2024 (UTC)[reply]
To add on, ( I'm sorry, I'm new, not trying to come off as rude, if I do please correct me )
Though only one section is marked as non-encyclopedic, reading through the article, there's a lot of strange analogies put as a fact.
I was wondering if it would be appropriate to just put the negatives in a separate criticism section? As a psychology student and APA member, I can say pretty confidently that overall in the scientific world, MBTI is seen as at least moderately accurate. Jasperthejas (talk) 20:23, 26 May 2024 (UTC)[reply]
No, the practice on Wikipedia is not to divide criticism into a separate section, that has been found to create large neutrality problems. And with respect, on Wikipedia the practice is to follow what is in reliable, independent sources, not your personal opinion as a psychology student. MrOllie (talk) 21:03, 26 May 2024 (UTC)[reply]
Barnum statements are often seen as at least moderately accurate, but that wouldn't be enough to undermine valid criticism. As they say, 'accurate' is not the same as 'precise', and neither are synonymous with 'useful'. Grayfell (talk) 23:14, 26 May 2024 (UTC)[reply]
On the other hand, Mr. @Grayfell's assessment that MBTI uses Barnum effect is based on a media source (not a journalistic source) without any critical data backing it off. Here's the whole statement from your cited source:
"Oddly enough, people are so willing to believe anything about their personalities that they'll fall for even the lamest explanations. In what's called the Barnum Effect, psychologists show that many people will fall (become "suckers") for generic personality explanations such as horoscopes and magazine self-tests as well as supposedly "scientific" tests administered by an "expert." "
For a full citation, the next paragraph is just a hand-waving statement, which could be contested:
"A good personality test can take what you’ve said and turn it to a useful analysis that gives you insights you didn’t have before. However, that analysis has to be based on sound statistical methods. The MBTI, at this point in time, is not." NgHanoi (talk) 03:11, 29 May 2024 (UTC)[reply]
My comment about MBTI using Barnum statements was intended to explain the issue. I was not citing any particular source for this. I don't know what you mean by the source being "a media source". The author is Susan Krauss Whitbourne, who is a Professor Emerita of Psychological and Brain Sciences at the University of Massachusetts Amherst.
If you want a journalistic source, here's "What Is MBTI: Is the Myers-Briggs Test Still Valid?" published in Discover Magazine a few months ago:
“Tests like the MBTI help us to organize our self-perceptions and experiences into coherent wholes,” says Stephen Benning, director of the Psychophysiology of Emotion and Personality Laboratory at the University of Nevada, Las Vegas. “The MBTI promises rich narratives to describe a respondent's personality, whereas other descriptions may feel more fragmented.”
Even though personality profiles are generic, people often feel that the descriptions accurately portray the image they have of themselves. This fallacy of personal validation is called the Barnum effect, which describes a person’s tendency to believe that vague personality descriptions are specifically tailored to them. The psychological phenomenon may also explain the appeal of astrology and fortune-telling.
Yes, any of this could be contested, but it has to be directly contested by reliable sources, not individual editors. Grayfell (talk) 19:52, 29 May 2024 (UTC)[reply]

Scientific vs. journalistic sources

[edit]

A lot of sources from the media are cited as validating references and they are mixed with scientific sources. I suggest to create a new section called "MBTI in the media" to group those journalistic sources, so that important topic like Validation and Accuracy are only sourced from the academic references. — Preceding unsigned comment added by NgHanoi (talkcontribs) 07:45, 17 May 2024 (UTC)[reply]

Article reads like a hit piece

[edit]

It reads like a hit piece against MBTI from the very outset. Even if you disagree with MBTI, there's no way you can look at the article in its current state and call it NPOV or encyclopedic in tone. Yes, we know it's not completely scientific. Yes, we know it has issues. And yet, many people very strongly believe it works as at least a basic way of categorizing personality types despite that.

Do we really need a full paragraph in the introduction dedicated to tearing the thing apart before MBTI has chance to be explained? Shouldn't the critisism be limited to a sentence or two and the end of the intro mentioning the cristiscisms, and then expand upon them later in the article after an explanation about MBTI itself has a chance to be laid out?

Even the articles on astrology and enneagrams seem less aggressively hostile than this one does in it's current state while still manage that each concept has detractors. It really seems like a certain population of editors has some sort of hard on in particular for ripping MBTI to shreds, and it really doesn't seem appropriate on a medium that's attempting to maintain an encyclopedic tone. Mr. UnderhiIl (talk) 12:04, 18 May 2024 (UTC)[reply]

Have a read of WP:YESBIAS. When the mainstream sources are critical, so to will be the Wikipedia article. That is what Wikipedia means by NPOV. MrOllie (talk) 12:50, 18 May 2024 (UTC)[reply]
Doesn't the fact that "50 million people have taken the Myers-Briggs Type Indicator and that 10,000 businesses, 2,500 colleges and universities, and 200 government agencies" kind of belie the idea that all "mainstream" sources are critical of MBTI? Mr. UnderhiIl (talk) 13:01, 18 May 2024 (UTC)[reply]
The "mainsteam" sources are academic publishers and peer-reviewed journals (and in this case, the ones _not_ published by the Myers–Briggs Foundation). Nearly every newspaper prints horoscopes, but we do not include those sources in our coverage of the solar system. MrOllie (talk) 13:05, 18 May 2024 (UTC)[reply]
Honestly, that really sounds like the main issue. It's not supposed to be some perfect, infallible measurement of personality that can provide repeatable results. It's a tool of self analysis. If you try to apply clinical standards to it, you're going to have people messing up the results for all sorts of reasons: they don't understand the purpose of the test; they try to skew it to some outcome, for various reasons, including wanting a "good" result or just to mess with the test givers for whatever perverse reason; maybe you just test them on a bad day when they aren't interested in answering accurately. The whole premise is ridiculous.
Should criticism be leveled towards its shortcomings and wrong applications (trying to use it for hiring/firing decisions, ignoring the possibilities for manipulation/reporting errors)? Sure.
I guess I'm not really sure how to incorporate what I'm saying here into the article, but I wish that the stance taken towards MBTI was more moderate from both critics and supporters. If there was way of conveying its uses in self-analysis by individuals engaged in the results with honesty, that'd be great. As it stands, someone reading this article with no knowledge of the subject would be pretty put off by the opening paragraphs, and that just seems wrong to me, as many people, including myself, would say it has been helpful to them. (P.S. Katharine Cook Briggs and Isabel Briggs Myers are dead, so if the foundation in their name is trying to misuse what they created, is it really their fault? Perhaps a rhetorical question, I know...) Mr. UnderhiIl (talk) 13:31, 18 May 2024 (UTC)[reply]
It sounds like you disagree with what the sources are saying, but Wikipedia really does not evaluate things that way. For better or worse, this is a site that exists to summarize what the independent experts have to say about a topic. Sometimes one might suspect that the expert position is leaving something out or is otherwise misguided, but nothing can be done to fix that here. MrOllie (talk) 13:48, 18 May 2024 (UTC)[reply]
MrOllie, it would be helpful in this debate if participants actually read their opponents' comments carefully and kept an open mind. You are asserting objectivity where there is none, and pretend to be speaking in the name of the whole Wikipedia project to support your personal stance.
This article *is* particularly biased, and its tone is unfit for Wikipedia. I was equally off-put by it, as I just wanted to read on the history of this (obviously rather unscientific) personality test, which is quite popular all over the world and has had a large impact on culture.
The tone and structure of this article (and this has nothing to do with whatever sources are cited) is that of one about a dangerous cult that mutilates children, not of one about a silly test people take to find some meaning in or make team-building decisions. Laughing Vampire (talk) 17:37, 2 July 2024 (UTC)[reply]
I'm not asserting 'objectivity'. I'm reiterating that when the independent sources are critical, so too will be the Wikipedia article. The sources are where the tone and structure come from - the sources are where everything comes from on Wikipedia. That is how this site works. MrOllie (talk) 18:14, 2 July 2024 (UTC)[reply]
The article uses direct language to explain that this test is pseudoscience in unambiguous terms. This is appropriate for an encyclopedia. The article doesn't imply anything about "mutilating children" and that comparison is not helpful.
To put it another way: many pseudoscientific things can be both popular and have an impact on culture. Wikipedia is not a popularity contest, it is an encyclopedia, so a neutral summary of a popular form of pseudoscience will still describe it as pseudoscience in direct language. The use of evasive language or euphemisms would make the article less neutral. As an encyclopedia, Wikipedia has an inherent 'bias' towards science and against pseudoscience, because our goal is to provide information and dispel misconceptions and misinformation. Grayfell (talk) 18:53, 2 July 2024 (UTC)[reply]

Previously cite-bombed source in lead

[edit]

@NgHanoi: This is the place to discuss your proposed changed. You have already edited this page, so I assume you still have the ability to edit it now.

For context, I removed this source from the article:

The source isn't inherently bad, but it's worth a closer look. It's a meta-analysis of three studies from 1977, 1986, and 2006. The study's authors emphasize over-and-over again that this set is, at best, barely sufficient for a meta-analysis.

It appears it was first added as part of a 'cite-bomb' in 2022 which was added to support MBTI's pseudoscientific status. This was then whittled down as a compromise. The source may be usable for something, but it isn't so valuable that it must be preserved. It's also not particularly helpful to demonstrating the consensus that MBTI is pseudoscience, so what, exactly, is it doing in this article?

Here are the other sources which were added for that cite-bomb:[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21]

References

  1. ^ Carlson, John G. (1985-08-01). "Recent Assessments of the Myers-Briggs Type Indicator". Journal of Personality Assessment. 49 (4): 356–365. doi:10.1207/s15327752jpa4904_3. ISSN 0022-3891. PMID 3900330.
  2. ^ Boyle, Gregory J. (1995-03). "Myers-Briggs Type Indicator (MBTI): Some Psychometric Limitations". Australian Psychologist. 30 (1): 71–74. doi:10.1111/j.1742-9544.1995.tb01750.x. {{cite journal}}: Check date values in: |date= (help)
  3. ^ Bess, Tammy L.; Harvey, Robert J. (2002-02-01). "Bimodal Score Distributions and the Myers-Briggs Type Indicator: Fact or Artifact?". Journal of Personality Assessment. 78 (1): 176–186. doi:10.1207/S15327752JPA7801_11. ISSN 0022-3891. PMID 11936208.
  4. ^ "APA PsycNet". psycnet.apa.org. Retrieved 2022-12-31.
  5. ^ Furnham, Adrian (1990-01-01). "The fakeability of the 16 PF, Myers-Briggs and FIRO-B personality measures". Personality and Individual Differences. 11 (7): 711–716. doi:10.1016/0191-8869(90)90256-Q. ISSN 0191-8869.
  6. ^ Lloyd, John B. (2012-04-01). "The Myers-Briggs Type Indicator® and mainstream psychology: analysis and evaluation of an unresolved hostility". Journal of Beliefs & Values. 33 (1): 23–34. doi:10.1080/13617672.2012.650028. ISSN 1361-7672.
  7. ^ "The Myers-Briggs Type Indicator and OD: Implication for Practice from Research - ProQuest". www.proquest.com. Retrieved 2022-12-31.
  8. ^ "The Rise Of The Myers-Briggs, Chapter 1: Katharine | Science Diction". WNYC Studios. Retrieved 2022-12-31.
  9. ^ Stromberg, Joseph (2014-07-15). "Why the Myers-Briggs test is totally meaningless". Vox. Retrieved 2022-12-31.
  10. ^ "Myers-Briggs Type Indicator: A Cultural and Ethical Evaluation". {{cite journal}}: Cite journal requires |journal= (help)
  11. ^ Murray, John B. (1990-06). "Review of Research on the Myers-Briggs Type Indicator". Perceptual and Motor Skills. 70 (3_suppl): 1187–1202. doi:10.2466/pms.1990.70.3c.1187. ISSN 0031-5125. {{cite journal}}: Check date values in: |date= (help)
  12. ^ Douglass, Frazier M.; Douglass, Robin (1993). "The Validity of the Myers-Briggs Type Indicator for Predicting Expressed Marital Problems". Family Relations. 42 (4): 422–426. doi:10.2307/585343. ISSN 0197-6664.
  13. ^ "APA PsycNet". psycnet.apa.org. Retrieved 2022-12-31.
  14. ^ Carskadon, Thomas G. (1977-12). "Test-Retest Reliabilities of Continuous Scores on the Myers-Briggs Type Indicator". Psychological Reports. 41 (3): 1011–1012. doi:10.2466/pr0.1977.41.3.1011. ISSN 0033-2941. {{cite journal}}: Check date values in: |date= (help)
  15. ^ Cohen, David; Cohen, Marilye; Cross, Herbert (1981-10). "A Construct Validity Study of the Myers-Briggs Type Indicator". Educational and Psychological Measurement. 41 (3): 883–891. doi:10.1177/001316448104100331. ISSN 0013-1644. {{cite journal}}: Check date values in: |date= (help)
  16. ^ Corman, Lawrence S.; Platt, Richard G. (1988-04). "Correlations among the Group Embedded Figures Test, the Myers-Briggs Type Indicator and Demographic Characteristics: A Business School Study". Perceptual and Motor Skills. 66 (2): 507–511. doi:10.2466/pms.1988.66.2.507. ISSN 0031-5125. {{cite journal}}: Check date values in: |date= (help)
  17. ^ Yancey-Bragg, N'dea. "Here's why people still take the Myers-Briggs test — even though it might not mean anything". USA TODAY. Retrieved 2022-12-31.
  18. ^ "APA PsycNet". psycnet.apa.org. Retrieved 2022-12-31.
  19. ^ "APA PsycNet". psycnet.apa.org. Retrieved 2022-12-31.
  20. ^ McCrae, Robert R.; Costa, Paul T. (1989-03). "Reinterpreting the Myers-Briggs Type Indicator From the Perspective of the Five-Factor Model of Personality". Journal of Personality. 57 (1): 17–40. doi:10.1111/j.1467-6494.1989.tb00759.x. ISSN 0022-3506. {{cite journal}}: Check date values in: |date= (help)
  21. ^ Pittenger, David J. (1993-12). "The Utility of the Myers-Briggs Type Indicator". Review of Educational Research. 63 (4): 467–488. doi:10.3102/00346543063004467. ISSN 0034-6543. {{cite journal}}: Check date values in: |date= (help)

Perhaps some of them are useful, but we need to look at what they are actually saying and evaluate them in that context. Grayfell (talk) 19:40, 21 May 2024 (UTC)[reply]

Mr.@Grayfell, thank you for your response. I intended to write a lengthy response but my guess is that you guys won't bother (Significant correlation with Big Five, researches confirming the Validity and Reliability existed, the historical perspective - MBTI existed long before Karl Popper and Factor analysis etc. will just be tossed out of the window)
The thing is that "the consensus that MBTI is pseudoscience" is just your thinking and my personal opinion MBTI is not pseudoscience (certainly based on peer-reviewed sources). As I looked up on Wikipedia's guide, at worst, MBTI might be classified (by Wikipedia's standard) as a Questionable science (WP:FRINGE/QS):
"Questionable science
Articles about hypotheses that have a substantial following but which critics describe as pseudoscience, may note those critics' views; however, such hypotheses should not be described as unambiguously pseudoscientific if a reasonable amount of academic debate still exists" NgHanoi (talk) 03:33, 29 May 2024 (UTC)[reply]
This doesn't really address my comment. If you wish to discuss these specific sources, do so. Otherwise, there are already many discussions on this talk page of why MBTI is regarded as pseudoscientific by the mainstream. But since you mention it, I will clarify that factor analysis was first applied to psychology decades before the MBTI was formalized during WWII. I'm not sure why that would even matter, though. Pseudoscience does not get an exemption for being old. Phrenology, for example, is a pseudoscience now, and was a pseudoscience when it was first popularized a century before Popper or MBTI. Popper's writing may or may not help reliable sources to identify pseudoscience, but we would have to look at those sources directly. Wikipedia does not publish original research.
As an aside, you do not know me well enough to know if I am a "Mr." or not. Calling me "Mr. Grayfell" is presumptuous. I don't take offense, but others might. Grayfell (talk) 19:36, 29 May 2024 (UTC)[reply]
I attempted to remove the "pseudoscience" wording from the very first sentence on two separate occasions a long time ago, at one point succeeding for a few months until more sources were scrapped together, but I won't retread any more of that now. I just wanted to point out the extreme opposition that has been repeatedly demonstrated towards the unambiguous classification of the MBTI as pseudoscience in the very first sentence of the lede. This opposition has been shrugged off repeatedly with poorly argued responses that make clear the original posts were not read in full, nor taken seriously. What we have here is a small vanguard of highly active Wikipedia nobility stonewalling against the actual consensus that has repeatedly been reached by those outside of this very limited oligarchy. I would advise everyone to not waste further effort pursuing changes to the article in opposition to this oligarchy. You will not succeed.
- PillageMe (talk) 11:50, 6 August 2024 (UTC)[reply]
I think it is exaggerated to describe the MBTI as pseudoscientific in the first sentence of the lead, and due to the lengthy and recurring debates I am going to post it on WP:NPOV/N. Vells (talk) 09:28, 21 August 2024 (UTC)[reply]
See Wikipedia:Neutral_point_of_view/Noticeboard#Myers–Briggs_Type_Indicator. --Vells (talk) 08:59, 24 August 2024 (UTC)[reply]